wehub-resource-sync
9201ef759e
4.3 KiB
Pydantic AI Provider Parity Explore
Model integrations live in
pydantic_ai_slim/pydantic_ai/models/ and …/providers/.
Objective
This is an explore, not a bug hunt. Pick one cross-cutting capability per run and map its support across all providers, surfacing silent gaps and inconsistencies a user would hit when switching providers.
Rotate the capability based on git log -1 --format=%cd --date=format:%j
modulo this list:
- Thinking / reasoning (
reasoning_effort, thinking parts, budgets). - Builtin tools (web search, web fetch, code execution) — presence & version.
- Usage & cost accounting (cached tokens, reasoning tokens, request counts).
- Structured output (native JSON schema / strict mode / tool-output mode).
- Streaming feature parity (deltas for thinking, tool args, usage on finish).
- Multimodal inputs (image / audio / document) per provider.
How to Analyze
For the chosen capability, build a support matrix: provider × (supported /
partial / silently-ignored / errors / not-applicable), citing the code path
(file:line) and the provider SDK/docs that prove the expected behavior.
Distinguish silent drops (input accepted, quietly ignored — a bug) from
explicit non-support (clearly raised / documented — acceptable).
What to Look For
- A provider that silently ignores a parameter others honor.
- Stale provider SDK pinning that omits a now-standard capability.
- Inconsistent defaults or types for the same conceptual feature.
- A capability documented as general but missing for a major provider.
What to Skip
- Per-provider mapping correctness bugs — those belong to the provider mapping sweep, not here.
- Speculative "would be nice" features with no user impact.
- Gaps already tracked by an open issue — search issues first.
Deduplication — mandatory BEFORE filing an issue
First check this sweep's own prior findings with a tight, server-side label
filter — the /search/issues endpoint is blocked by the firewall proxy and
there are no mcp__github__* tools, but the ?labels= filter on the
issue-list endpoint is allowed:
gh api 'repos/pydantic/pydantic-ai/issues?state=open&labels=provider-parity-explore&per_page=100' \
--jq '.[] | select(.pull_request == null) | {number, title}'
Only if that is inconclusive, widen to a full open-issue scan and grep locally for "parity" and the capability you're auditing:
gh api --paginate 'repos/pydantic/pydantic-ai/issues?state=open&per_page=100' \
--jq '.[] | select(.pull_request == null) | {number, title, labels: [.labels[].name]}'
If a matching issue exists, call mcp__safeoutputs__noop immediately.
Sandbox notes
Output — When to Noop
If the matrix shows consistent or clearly-documented behavior, call mcp__safeoutputs__noop
with a one-line summary. Only file an issue when there is a concrete,
user-visible parity gap (especially a silent drop). At most one issue per
run.
Issue Format
Title: Provider parity: <capability> — <gap summary>
Body:
Capability
[What was audited this run]
Support Matrix
Provider Status Code path Notes … supported / partial / silently-ignored / errors file:line… Concrete Gap
[The specific user-visible problem and which provider(s)]
Evidence
- [SDK/docs references;
path:line; a short snippet showing the silent drop]Adversarial review
- Reproduced on
main: [exact command + real output demonstrating the gap]- Existing tests checked: [tests read; none assert this behavior is intentional]
- Ruled out by-design: [nearby comment / profile / maintainer decision checked]
- SDK verified for this provider: [the real type/shape, not inferred by analogy to another provider]
- Not a duplicate: [label-filtered dedup returned nothing]