# Provider conformance

This harness keeps the services → agents → workflows lifecycle honest across the
supported AI providers. It runs the same end-to-end scenarios against each
configured provider and treats missing provider keys as an explicit skip, so the
suite is safe for local development, forks, and scheduled CI.

## What it exercises

`go run ./internal/harness/provider-conformance` fans out over the provider-facing
agent test and the harnesses in `internal/harness`:

- `agent` — provider tool-call conformance through `agent.Ask`, including run metadata propagation.
- `universe` — service discovery plus agent tool calls over the real runtime.
- `agent-flow` — a workflow event that drives an agent to call services.
- `plan-delegate` — plan persistence plus agent-to-agent delegation and service
  calls.
- `a2a-stream-fallback` — A2A `message/stream` through the gateway, including
  fallback from unsupported provider streaming to the tool-calling `Ask` path while
  preserving run metadata.

The command also emits the registered provider capability matrix so the run shows
which providers advertise model, image, video, and streaming support. Console output
and summary artifacts label each harness with the phase it is proving (for example,
model call + tool call, workflow event + tool call, or streaming fallback + tool
call), so provider failures identify the failed lifecycle phase instead of only the
provider name.

## Local usage

Run the deterministic path with no secrets:

```sh
go run ./internal/harness/provider-conformance -providers mock
```

Run every live provider that has a key in the environment:

```sh
go run ./internal/harness/provider-conformance \
  -summary-json provider-conformance-summary.json \
  -summary-markdown provider-conformance-summary.md \
  -capabilities-markdown provider-capabilities.md
```

Provider keys are read from `MICRO_AI_API_KEY` or the provider-specific variable:

| Provider | Secret / environment variable |
| --- | --- |
| Anthropic | `ANTHROPIC_API_KEY` |
| OpenAI | `OPENAI_API_KEY` |
| Gemini | `GEMINI_API_KEY` |
| Groq | `GROQ_API_KEY` |
| MiniMax | `MINIMAX_API_KEY` |
| Mistral | `MISTRAL_API_KEY` |
| Together | `TOGETHER_API_KEY` |
| AtlasCloud | `ATLASCLOUD_API_KEY` |

Use `-require-configured` when you want a selected provider without a key to fail
instead of skip. This is useful for manually checking that a required provider
secret is actually wired into CI before relying on that provider as covered:

```sh
go run ./internal/harness/provider-conformance \
  -providers anthropic,openai \
  -require-configured
```

## Scheduled CI behavior

The `Harness (E2E)` workflow runs on pushes and pull requests with deterministic
mock LLMs, including `provider-conformance -providers mock`. On the hourly
schedule and manual dispatch it also runs the live provider conformance job. A
manual dispatch can narrow `providers` or `harnesses`, and can set
`require_configured=true` to fail fast when an expected repository secret is
missing; scheduled runs keep the safe default and report missing keys as skips.
That job:

1. runs the same `agent`, `universe`, `agent-flow`, `plan-delegate`, and `a2a-stream-fallback` harness list,
2. reads the provider keys from repository secrets,
3. skips providers whose secrets are absent,
4. fails when any configured provider fails a harness, and
5. uploads JSON and Markdown coverage artifacts for the run.

The job also appends the Markdown summary and capability matrix to the GitHub
Actions step summary, making configured, skipped, and failed provider coverage
visible without downloading artifacts.

## Adding a provider

To bring a new provider into scheduled conformance:

1. register its `ai` provider implementation and capability metadata,
2. add the provider name and key variable to `providerEnv` in `main.go`,
3. import the provider package in `main.go`,
4. pass the matching repository secret through `.github/workflows/harness.yml`,
   and
5. run `go run ./internal/harness/provider-conformance -providers <name> \
   -require-configured` with a live key before opening the change.
