pydantic--pydantic-ai/.github/workflows/shared/prompts/pydantic-ai-regression-detector.md

Pydantic AI Regression Detector

Objective

Find one behavioral regression — something that worked in a recent released version and broke in a later one. Regressions are the most upgrade-blocking bug class for users.

Data Gathering

1. Identify the two most recent release tags: git tag --sort=-v:refname (fall back to git log --tags). Call them OLD and NEW.
2. git log --oneline OLD..NEW -- pydantic_ai_slim and read diffs of changes with user-facing surface: public Agent API, run/run_stream/iter, message-history semantics, output/result_type validation, provider request/response mapping, tool dispatch, usage accounting.
3. Optionally scan open issues for "worked in", "regression", "after upgrading", "broke in" to corroborate — but you must still reproduce it yourself.

How to Verify — mandatory

Write a new minimal test that exercises the behavior. Demonstrate it passes on OLD and fails on NEW — e.g. git stash/worktree or uv run --with 'pydantic-ai-slim==<OLD>' vs the working tree. A change that only "looks risky" in the diff is not a finding.

What to Look For

• Changed defaults, exceptions, or error messages users depend on.
• Output/validation behavior change for the same inputs.
• Message-history / new_messages() shape or re-feed semantics changing.
• Provider mapping that regressed for a previously-working call.
• Behavioral/semantic changes that break documented usage patterns.

What to Skip

• Intentional, documented breaking changes (check CHANGELOG / release notes / v2-labeled work) — those are not regressions.
• Speculation without an old-passes/new-fails reproduction.
• Behavior already tracked by an open issue — search issues first.

Deduplication — mandatory BEFORE filing an issue

First narrow to regression-labelled issues with a tight, server-side filter — the /search/issues endpoint is blocked by the firewall proxy and there are no mcp__github__* tools, but the ?labels= filter on the issue-list endpoint is allowed. This covers both prior [regression-detector] findings and human-filed regression reports:

gh api 'repos/pydantic/pydantic-ai/issues?state=open&labels=regression&per_page=100' \
  --jq '.[] | select(.pull_request == null) | {number, title}'

Only if that is inconclusive, widen to a full open-issue scan and grep locally for "regression" and the affected symbol:

gh api --paginate 'repos/pydantic/pydantic-ai/issues?state=open&per_page=100' \
  --jq '.[] | select(.pull_request == null) | {number, title, labels: [.labels[].name]}'

If a matching issue exists, call mcp__safeoutputs__noop immediately.

Sandbox notes

• Use the native Grep tool for codebase search.

Quality Gate — When to Noop

mcp__safeoutputs__noop is the expected outcome most runs. Only file when you have a concrete test that passes on OLD and fails on NEW, with both outputs captured.

Issue Format

Title: Regression: <behavior> broke between <OLD> and <NEW>

Body:

Impact

[Who is affected on upgrade and how]

Versions

Last working: OLD · First broken: NEW Suspected commit(s): [SHA(s) with links]

Reproduction

[The new test — full code — and exact commands for OLD and NEW]

Expected vs Actual

OLD output: … NEW output: …

Evidence

• [Captured outputs for both versions; path:line of the change]

Adversarial review

• Reproduced on OLD and NEW: [commands + real outputs for both versions]
• Change was NOT intentional: [found the commit/PR that changed it and confirmed it wasn't a deliberate behavior change — the #1 reason regression reports get rejected]
• Existing tests checked: [NEW-version tests read; none assert the new behavior as intended]
• Not a duplicate: [label-filtered dedup returned nothing]