# Agent provider conformance matrix

`go test ./...` includes `TestAgentProviderConformanceMatrix`, a shared agent
scenario that runs against every registered chat provider. The scenario asks an
agent to call a deterministic local tool, verifies the tool receives `ai.RunInfo`,
and checks the final response carries the conformance marker. The live matrix
includes MiniMax in the tool/guardrail path in addition to providers with
streaming coverage, so every supported chat provider has at least one key-gated
agent contract. A fake provider path runs on every machine without network
access so CI always exercises the harness.

Live providers are opt-in to avoid flaky unauthenticated PR checks and accidental
API spend. To run the live matrix, set `GO_MICRO_AGENT_CONFORMANCE_LIVE=1` plus the
provider API keys you want to exercise:

| Provider | Required API key | Optional model override |
| --- | --- | --- |
| OpenAI | `OPENAI_API_KEY` | `GO_MICRO_CONFORMANCE_OPENAI_MODEL` |
| Anthropic | `ANTHROPIC_API_KEY` | `GO_MICRO_CONFORMANCE_ANTHROPIC_MODEL` |
| Atlas Cloud | `ATLASCLOUD_API_KEY` | `GO_MICRO_CONFORMANCE_ATLASCLOUD_MODEL` |
| Gemini | `GEMINI_API_KEY` | `GO_MICRO_CONFORMANCE_GEMINI_MODEL` |
| Groq | `GROQ_API_KEY` | `GO_MICRO_CONFORMANCE_GROQ_MODEL` |
| MiniMax | `MINIMAX_API_KEY` | `GO_MICRO_CONFORMANCE_MINIMAX_MODEL` |
| Mistral | `MISTRAL_API_KEY` | `GO_MICRO_CONFORMANCE_MISTRAL_MODEL` |
| Together | `TOGETHER_API_KEY` | `GO_MICRO_CONFORMANCE_TOGETHER_MODEL` |

When `GO_MICRO_AGENT_CONFORMANCE_LIVE` or a provider key is absent, the live
provider subtest reports a deterministic skip. When both are present, a provider
failure is a real test failure because drift in chat, tool calling, run metadata,
or final-answer behavior means the services → agents lifecycle is no longer
consistent across providers.

The companion `TestAgentProviderConformanceFakeError` keeps provider error
propagation covered locally without relying on external credentials.

## Local no-secret conformance

Use `make provider-conformance-mock` to run the same provider conformance harness
through the deterministic mock provider. That target requires no API keys and is
what `make harness` delegates to after the 0→1 and 0→hero scenarios, so every PR
continues to exercise the provider-facing agent/tool contract without spending
live model credits.

Use `make provider-conformance` when you want the live-provider sweep: providers
without keys are skipped, and configured providers must satisfy the same harness
contract. For the exact scheduled command, run:

```sh
go run ./internal/harness/provider-conformance \
  -providers anthropic,openai,gemini,groq,minimax,mistral,together,atlascloud \
  -harnesses agent,universe,agent-flow,plan-delegate,a2a-stream-fallback \
  -summary-json provider-conformance-summary.json \
  -summary-markdown provider-conformance-summary.md \
  -capabilities-markdown provider-capabilities.md
```

The generated summary records one row for every selected provider/harness pair;
missing live-provider keys become skipped rows for each harness, while configured
providers produce pass/fail rows per harness.

## Scheduled CI

The hourly/manual `Harness (E2E)` workflow runs the same matrix with
`GO_MICRO_AGENT_CONFORMANCE_LIVE=1` and the provider secrets exported. Providers
whose keys are absent still skip cleanly, while any configured provider must pass
the shared tool-calling scenario. This keeps scheduled conformance key-gated: PR
checks stay deterministic and no-key environments remain green, but maintained
provider credentials exercise the live matrix regularly.