wehub-resource-sync
e071084ebe
govulncheck / govulncheck (push) Has been cancelled
Lint / golangci-lint (push) Has been cancelled
Run Tests / Unit Tests (push) Has been cancelled
Run Tests / Etcd Integration Tests (push) Has been cancelled
Harness (E2E) / Harnesses (mock LLM) (push) Has been cancelled
Harness (E2E) / Provider harnesses (live LLM conformance) (push) Has been cancelled
69 lines
3.5 KiB
Markdown
69 lines
3.5 KiB
Markdown
# Agent provider conformance matrix
|
|
|
|
`go test ./...` includes `TestAgentProviderConformanceMatrix`, a shared agent
|
|
scenario that runs against every registered chat provider. The scenario asks an
|
|
agent to call a deterministic local tool, verifies the tool receives `ai.RunInfo`,
|
|
and checks the final response carries the conformance marker. The live matrix
|
|
includes MiniMax in the tool/guardrail path in addition to providers with
|
|
streaming coverage, so every supported chat provider has at least one key-gated
|
|
agent contract. A fake provider path runs on every machine without network
|
|
access so CI always exercises the harness.
|
|
|
|
Live providers are opt-in to avoid flaky unauthenticated PR checks and accidental
|
|
API spend. To run the live matrix, set `GO_MICRO_AGENT_CONFORMANCE_LIVE=1` plus the
|
|
provider API keys you want to exercise:
|
|
|
|
| Provider | Required API key | Optional model override |
|
|
| --- | --- | --- |
|
|
| OpenAI | `OPENAI_API_KEY` | `GO_MICRO_CONFORMANCE_OPENAI_MODEL` |
|
|
| Anthropic | `ANTHROPIC_API_KEY` | `GO_MICRO_CONFORMANCE_ANTHROPIC_MODEL` |
|
|
| Atlas Cloud | `ATLASCLOUD_API_KEY` | `GO_MICRO_CONFORMANCE_ATLASCLOUD_MODEL` |
|
|
| Gemini | `GEMINI_API_KEY` | `GO_MICRO_CONFORMANCE_GEMINI_MODEL` |
|
|
| Groq | `GROQ_API_KEY` | `GO_MICRO_CONFORMANCE_GROQ_MODEL` |
|
|
| MiniMax | `MINIMAX_API_KEY` | `GO_MICRO_CONFORMANCE_MINIMAX_MODEL` |
|
|
| Mistral | `MISTRAL_API_KEY` | `GO_MICRO_CONFORMANCE_MISTRAL_MODEL` |
|
|
| Together | `TOGETHER_API_KEY` | `GO_MICRO_CONFORMANCE_TOGETHER_MODEL` |
|
|
|
|
When `GO_MICRO_AGENT_CONFORMANCE_LIVE` or a provider key is absent, the live
|
|
provider subtest reports a deterministic skip. When both are present, a provider
|
|
failure is a real test failure because drift in chat, tool calling, run metadata,
|
|
or final-answer behavior means the services → agents lifecycle is no longer
|
|
consistent across providers.
|
|
|
|
The companion `TestAgentProviderConformanceFakeError` keeps provider error
|
|
propagation covered locally without relying on external credentials.
|
|
|
|
## Local no-secret conformance
|
|
|
|
Use `make provider-conformance-mock` to run the same provider conformance harness
|
|
through the deterministic mock provider. That target requires no API keys and is
|
|
what `make harness` delegates to after the 0→1 and 0→hero scenarios, so every PR
|
|
continues to exercise the provider-facing agent/tool contract without spending
|
|
live model credits.
|
|
|
|
Use `make provider-conformance` when you want the live-provider sweep: providers
|
|
without keys are skipped, and configured providers must satisfy the same harness
|
|
contract. For the exact scheduled command, run:
|
|
|
|
```sh
|
|
go run ./internal/harness/provider-conformance \
|
|
-providers anthropic,openai,gemini,groq,minimax,mistral,together,atlascloud \
|
|
-harnesses agent,universe,agent-flow,plan-delegate,a2a-stream-fallback \
|
|
-summary-json provider-conformance-summary.json \
|
|
-summary-markdown provider-conformance-summary.md \
|
|
-capabilities-markdown provider-capabilities.md
|
|
```
|
|
|
|
The generated summary records one row for every selected provider/harness pair;
|
|
missing live-provider keys become skipped rows for each harness, while configured
|
|
providers produce pass/fail rows per harness.
|
|
|
|
## Scheduled CI
|
|
|
|
The hourly/manual `Harness (E2E)` workflow runs the same matrix with
|
|
`GO_MICRO_AGENT_CONFORMANCE_LIVE=1` and the provider secrets exported. Providers
|
|
whose keys are absent still skip cleanly, while any configured provider must pass
|
|
the shared tool-calling scenario. This keeps scheduled conformance key-gated: PR
|
|
checks stay deterministic and no-key environments remain green, but maintained
|
|
provider credentials exercise the live matrix regularly.
|