wehub/openai--openai-agents-python
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:39:17 +08:00
commit 4ed4e9ff99
1368 changed files with 334957 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/runtime-behavior-probe/SKILL.md
+161
View File
@@ -0,0 +1,161 @@
---
name: runtime-behavior-probe
description: Plan and execute runtime-behavior investigations with temporary probe scripts, validation matrices, state controls, and findings-first reports. Use only when the user explicitly invokes this skill to verify actual runtime behavior beyond normal code-level checks, especially to uncover edge cases, undocumented behavior, or common failure modes in local or live integrations. A baseline smoke check is fine as an entry point, but do not stop at happy-path confirmation.
---
# Runtime Behavior Probe
## Overview
Use this skill to investigate real runtime behavior, not to restate code or documentation. Start by planning the investigation, then execute a case matrix, record observed behavior, and report both the findings and the method used to obtain them.
## Core Rules
- Treat this skill as manual-only. Do not rely on implicit invocation.
- A baseline success or smoke case is often the right entry point, but do not stop there when the real question involves edge cases, drift, or failure behavior.
- Plan before running anything. Write the case matrix first, then fill it in with observed results. The matrix can live in a scratch note, a temporary file, or the probe script header.
- Default to local or read-only probes. Consider a live service only when it is clearly relevant, then apply the lightweight gates below before you run it.
- Size the probe to the decision. Start with the smallest matrix that can disqualify or validate the current hypothesis, then expand only when uncertainty remains.
- Before a live probe, apply three lightweight gates:
- Destination gate. Use only a live destination that is clearly allowed for the task.
- Intent gate. Run the live probe only when the user explicitly wants runtime verification on that integration, or explicitly approves it after you propose the probe.
- Data gate. If the probe will read environment variables, mutate remote state, incur material cost, or exercise non-public or user data, name the exact variable names or data class and get explicit approval first.
- Classify each case as read-only, mutating, or costly before execution. For mutating or costly cases, or for any live case that will read environment variables, define cleanup or rollback before running the probe.
- Use temporary files or a temporary directory for one-off probe scripts.
- Keep temporary artifacts until the final response is drafted. Then delete them by default unless the user asked to keep them or they are needed for follow-up. Even when artifacts are deleted, keep a short run summary of the command shape, runtime context, and artifact status in the report.
- Before executing a live probe that will read environment variables, tell the user the exact variable names you plan to use and why, then wait for explicit approval. Examples include `OPENAI_API_KEY` and other expected default names for the system under test.
- When the environment-variable approval gate is required and the `request_user_input` tool is available, use that tool instead of a plain-text approval question. Ask one concise question with mutually exclusive choices such as `Allow once (Recommended)` and `Do not allow`, omit `autoResolutionMs`, and make the approval single-probe and limited to the exact named variables and destination. If the tool is unavailable, fall back to a concise plain-text approval question and do not proceed until the user explicitly approves.
- Never print secrets, even when they come from standard environment variables that this skill may use.
- For OpenAI API or OpenAI platform probes in this repository, use [$openai-knowledge](../openai-knowledge/SKILL.md) early to confirm contract-sensitive details such as supported parameters, field names, and limits. Use runtime probing to validate or challenge the documented behavior, not to skip the documentation pass entirely. If the docs MCP is unavailable, fall back to the official OpenAI docs and say that you used the fallback in the report.
- For benchmark or comparison probes, make parity explicit before execution. Record what is held constant, what variable is under test, which response-shape constraints keep the comparison fair, and any usage or token counters that matter for interpreting latency or cost.
- For OpenAI hosted tool probes, remove setup ambiguity before attributing a negative result to runtime behavior:
- Force the tool path with the matching `tool_choice` when the question depends on tool invocation.
- Treat `container_auto` and `container_reference` as separate cases, not interchangeable setup details.
- Clear unsupported model or tool options first so they do not invalidate the probe.
## Workflow
1. Restate the investigation target in operational terms. Name the runtime surface, the key uncertainty, and the highest-risk behaviors to test.
2. Do a short preflight. Check the relevant code or docs first, decide whether the question needs local or live validation, and note any repo, baseline, or release boundary that matters.
3. Create a validation matrix before executing probes. Cover both baseline behavior and the most relevant failure or drift cases. The matrix can live in a scratch note, a temporary file, or a structured header inside the probe script.
4. For each case, choose an execution mode up front:
- `single-shot` for deterministic one-run checks.
- `repeat-N` for cache, retry, streaming, interruption, rate-limit, concurrency, or other run-to-run-sensitive behavior.
- `warm-up + repeat-N` when first-run cold-start effects could distort the result.
Use these defaults unless the task clearly needs something else:
- Quick screen of a repeat-sensitive question: `repeat-3`.
- Decision-grade latency or release recommendation: `warm-up + repeat-10`.
- Costly live cases: start at `repeat-3`, then expand only if the answer remains unclear.
If it is genuinely unclear whether extra runs are worth the time or cost, ask the user before expanding the probe.
5. When the question is benchmark-like or comparative, run in phases. Start with a high-signal pilot matrix against a control, then expand only the surviving candidates or unresolved cases.
6. If the question is about a suspected regression or behavior change, add at least one known-good control case such as `origin/main`, the latest release, or the same request without the suspected option.
7. For comparative probes, define parity before execution. Record prompt or input shape, tool-choice setup, model-settings parity, state reuse rules, and any response-shape constraint that keeps the comparison fair. If materially different output length could bias the result, record usage or token notes too.
8. If the question asks whether one option has the same intelligence or quality as another, decide whether the matrix supports only example-pattern parity or a broader quality claim. For broader claims, add at least one harder or more open-ended case. Otherwise say explicitly that the result is limited to the covered patterns.
9. Plan state controls before execution when hidden state could affect the result. Record whether each case uses fresh or reused state, how cache reuse or cache busting is handled, what unique IDs isolate repeated runs, and how cleanup is verified.
10. If any live case will read environment variables, list the exact variable names and purpose for each case, then ask the user for approval before execution. Prefer `request_user_input` for this gate when it is available, with no auto-resolution and choices that grant or deny only this specific probe. Keep the approval ask short and include destination, read-only versus mutating or costly risk, exact variable names, and cleanup or rollback if relevant.
11. Build task-specific probe scripts in a temporary location. Keep the script small, observable, and easy to discard.
12. In `openai-agents-python`, make the runtime context explicit:
- Run Python probes from the repository root with `uv run python` when practical.
- Record the current commit, working directory, Python executable, and Python version.
- Avoid accidental imports from a different checkout or site-packages location. If you must deviate from `uv run python`, say exactly why and what interpreter or environment was used instead.
13. Execute the matrix and capture evidence. Record request shape, setup, observation summary, unexpected or negative result, error details, timing, runtime context, approved environment-variable names, repeat counts, warm-up handling, variance when relevant, cleanup behavior, and for comparisons note what was held constant plus any response-shape or usage notes that affect interpretation.
14. Update the matrix with actual outcomes, not guesses.
15. Keep temporary artifacts until the final response is drafted. Then delete them unless the user asked to keep them or they are needed for follow-up. Benchmark and repeat-heavy probes often need follow-up, so keeping artifacts is normal when the result may be revisited. If deleted, retain and report a short run summary.
16. Report findings first, with unexpected or negative findings first. Then summarize how the validation was performed and which cases were covered.
17. If the probe isolates one clear defect, you may include a short implementation hypothesis or minimal repro direction. Do not expand into a larger next-step plan unless the user asked for it.
## Validation Matrix
Use a matrix that makes the news easy to scan. Start from the runtime question and the observation summary, not just from `expected` and `pass` or `fail`.
Use a matrix with at least these columns:
- `case_id`
- `scenario`
- `mode`
- `question`
- `setup`
- `observation_summary`
- `result_flag`
- `evidence`
Add these columns when they materially improve the investigation:
- `comparison_basis`
- `variable_under_test`
- `held_constant`
- `output_constraint`
- `status`
- `confidence`
- `state_setup`
- `repeats`
- `warm_up`
- `variance`
- `usage_note`
- `risk_profile`
- `env_vars`
- `approval`
- `control`
Treat `result_flag` as a fast scan field such as `unexpected`, `negative`, `expected`, or `blocked`. Use `status` only when there is a credible comparison basis, baseline, or documented contract to compare against.
Always consider whether the matrix should include these categories:
- Baseline success.
- Control or baseline comparison when a regression is suspected.
- Boundary input or parameter variation.
- Invalid or unsupported input.
- Missing or incorrect configuration.
- Transient external failure such as timeout, network interruption, or rate limiting.
- Retry, idempotence, or cleanup behavior.
- Concurrency or overlapping operations when shared state or ordering may matter.
- Open-ended quality or intelligence samples when the question is broader than pattern parity.
Open [validation-matrix.md](./references/validation-matrix.md) when you need a stronger prioritization model or a reusable case template.
## Temporary Probe Scripts
Write one-off scripts in a temporary file or temporary directory such as one created by `mktemp -d` or Python `tempfile`. Keep the script outside the repository by default, even when it imports code from the repository.
If the probe needs repository code:
- Run it with the repository as the working directory, or
- Set `PYTHONPATH` or the equivalent import path explicitly.
- In `openai-agents-python`, prefer `uv run python /tmp/probe.py` from the repository root.
Design the probe to maximize observability:
- Print or log the exact scenario being exercised.
- Capture runtime context such as git SHA, working directory, Python executable and version, relevant package versions, model or deployment name, endpoint or base URL alias, and any retry or tool options that materially affect behavior.
- For live probes, record only the names of environment variables that were approved for use. Never print their values.
- Capture structured outputs when possible.
- Preserve raw error type, message, and status code.
- For repeat-sensitive cases, capture the attempt index, warm-up status, and any stable identifiers that help compare runs.
- For repeated or benchmark-style probes, write both raw results and a compact summary artifact when practical.
- Keep branching minimal so each script answers a narrow question.
Before deleting the temporary script or directory, keep a short run summary of the script path, command used, runtime context, and whether the evidence was kept or deleted.
Open [python_probe.py](./templates/python_probe.py) when you want a lightweight disposable Python probe scaffold.
## Reporting
Report in this order:
1. Findings. Put unexpected or negative findings first. If there was no real news, say that explicitly.
2. Validation approach. Summarize the code used, the runtime surface exercised, the execution modes, and the case matrix coverage.
3. Case results. Include the matrix or a condensed version of it when the case count is large.
4. Artifact status and brief run summary. State whether temporary artifacts were deleted or kept, and provide kept paths or the retained summary.
5. Optional implementation note. Include this only when one clear defect was isolated and a short implementation direction would help.
For comparative probes, the report should also say what was held constant, what variable was under test, and whether the result supports only pattern parity or a broader quality claim.
Open [reporting-format.md](./references/reporting-format.md) for the recommended response template.
## Resources
- Open [validation-matrix.md](./references/validation-matrix.md) to design and prioritize the case matrix.
- Open [error-cases.md](./references/error-cases.md) to expand common failure scenarios.
- Open [openai-runtime-patterns.md](./references/openai-runtime-patterns.md) for recurring OpenAI and Responses API probe patterns.
- Open [reporting-format.md](./references/reporting-format.md) for the final report structure.
- Open [python_probe.py](./templates/python_probe.py) for a minimal disposable Python probe scaffold.
.agents/skills/runtime-behavior-probe/agents/openai.yaml
+6
View File
@@ -0,0 +1,6 @@
interface:
display_name: "Runtime Behavior Probe"
short_description: "Plan and run runtime behavior probes"
default_prompt: "Use $runtime-behavior-probe to investigate actual runtime behavior with a validation matrix, explicit state controls, and a findings-first report."
policy:
allow_implicit_invocation: false
.agents/skills/runtime-behavior-probe/references/error-cases.md
+80
View File
@@ -0,0 +1,80 @@
# Common Error Cases
Use this reference to expand beyond the happy path. Favor error cases that a real user or operator is likely to hit.
## Configuration Errors
Check whether the runtime behaves differently for:
- Missing required environment variables.
- Present but malformed secrets or identifiers.
- Wrong endpoint or base URL.
- Wrong model or deployment name.
- Incompatible local dependency versions.
Look for:
- Error type and status code.
- Whether the failure is immediate or delayed.
- Whether the message is actionable.
- Whether retrying without fixing configuration changes anything.
## Input Errors
Probe common bad-input patterns such as:
- Missing required fields.
- Wrong data type.
- Unsupported enum or option value.
- Empty but syntactically valid input.
- Oversized input or too many items.
- Mutually incompatible options.
Prefer realistic invalid inputs over artificial nonsense. The point is to learn how the runtime fails in practice.
## Transport and Availability Errors
When networked services are involved, consider:
- Connection failure.
- Read timeout.
- Server timeout or upstream gateway error.
- Rate limit response.
- Partial stream interruption.
- Reusing a connection after a failure.
Capture whether the client library retries automatically, whether it surfaces retry metadata, and whether the final exception preserves the original cause.
## State and Repetition Errors
Many surprising bugs appear only when an operation is repeated or interrupted:
- Re-submit the same request.
- Repeat after a timeout.
- Retry after a partial tool call or partial stream.
- Resume after local cleanup or process restart.
- Repeat with slightly changed inputs while reusing shared state.
Observe whether the operation is idempotent, duplicated, silently ignored, or left in a partial state.
## Concurrency Errors
When shared state, ordering, or isolation may matter, consider:
- Two overlapping requests with the same logical input.
- Parallel runs that reuse the same cache key, session, container, or temporary resource.
- Concurrent retries, cancellation, or cleanup racing with active work.
- Output or event streams from one run leaking into another.
Capture whether the runtime serializes, rejects, duplicates, corrupts, or cross-contaminates the work.
## Investigation Heuristics
Use these heuristics to pick error cases quickly:
- Ask which failure a real engineer would debug first in production.
- Ask which failure is most expensive if it is misunderstood.
- Ask which failure would be invisible from code review alone.
- Ask which failure path is likely to differ across environments.
If the error behavior is already perfectly obvious from a local validator or type system, it is usually low priority for this skill.
.agents/skills/runtime-behavior-probe/references/openai-runtime-patterns.md
+137
View File
@@ -0,0 +1,137 @@
# OpenAI Runtime Patterns
Use this reference for recurring OpenAI investigations so you do not have to rediscover the probe strategy each time. In this repository, use [$openai-knowledge](../../openai-knowledge/SKILL.md) up front for contract-sensitive details, then use this reference to design the runtime validation. If the docs MCP is unavailable, fall back to the official OpenAI docs and say so in the report.
## General Rules
- Prefer small live probes over large harnesses.
- Keep one script focused on one uncertainty.
- For comparative or benchmark-like questions, start with a pilot and expand only when the answer is still unclear.
- Capture both the request shape and the returned item types.
- Preserve raw error payloads and status codes.
- Record whether behavior differs between the first call and a repeated call.
- When the question is about regression or contract drift, add a known-good control run before attributing the result to the change under investigation.
- Keep comparison parity explicit. Record what was held constant, what variable changed, and whether output-shape or usage differences could bias the conclusion.
- When the question depends on tool invocation, force the target path with the matching `tool_choice`.
- Treat `container_auto` and `container_reference` as distinct setup modes, not interchangeable details.
- Clear unsupported model or tool options before diagnosing runtime behavior.
## Standard Environment Variables
Do not read these variables automatically. Before a live probe uses any of them, tell the user the exact variable names you plan to read and why each one is needed, then wait for explicit approval. Never print their values:
- `OPENAI_API_KEY`
- `OPENAI_BASE_URL`
- `OPENAI_ORG_ID`
- `OPENAI_PROJECT_ID`
If the task targets another standard integration, use that integration's expected default variable names under the same rule.
## Environment False Signals
Before attributing a failure to the patch under review, exclude environment and source-selection problems with a control run.
- Confirm the commit and worktree under test. When editable installs, shared environments, `PYTHONPATH`, or generated artifacts can select stale code, verify the imported package path and rebuild before probing.
- Run base and head controls with the same interpreter, dependencies, environment variables, and command shape.
- Treat proxy initialization, sandbox denials, unavailable containers, expired snapshots, authentication, quotas, rate limits, service outages, and stale caches as environment conditions until a controlled rerun ties them to the patch.
- Never print proxy URLs or credentials. Change only the minimum in-scope environment or disposable state needed for the control run, and record which variable names or constraints changed.
In the final report, distinguish code failures, unsupported configurations, environment blockers, and inconclusive probes. Do not combine them into one failed-test count.
## Responses API Probe Patterns
For Responses API work, start from the uncertainty instead of from the full feature surface.
### Benchmark or model-switch comparisons
Use when you need to compare models, settings, transports, or providers with enough rigor to support a product or release decision.
Probe suggestions:
- Start with a pilot that includes one control and two or three highest-signal scenarios.
- Keep prompt shape, tool choice, state setup, and non-tested settings aligned across candidates.
- If the question is about speed, capture medians and, when relevant, first-token latency plus any usage note that could explain the difference.
- If the question is about "same intelligence" or "same quality," add at least one harder or more open-ended case. Otherwise report the result as pattern parity only.
- Expand to a larger matrix only when the pilot survives, the candidates are close, or a major runtime surface is still uncovered.
### Plain response behavior
Use when you need to confirm:
- The shape of returned output items.
- Whether text appears in one item or multiple items.
- How metadata appears in the final object.
Probe suggestions:
- Baseline call with a minimal input.
- Same call with a slightly different instruction shape.
- Repeat the same call to check output stability where that matters.
### Structured output behavior
Use when you need to observe:
- Schema rejection versus best-effort completion.
- Handling of missing required fields.
- Differences between model-compliant output and transport-level errors.
Probe suggestions:
- Valid schema and valid prompt.
- Prompt likely to produce omitted fields.
- Clearly incompatible schema or unsupported option when relevant.
### Tool invocation behavior
Use when you need to learn:
- When tool calls are emitted.
- How arguments are shaped at runtime.
- What happens when the tool fails or returns malformed output.
Probe suggestions:
- Baseline tool-call success.
- Tool failure with a realistic exception.
- Tool result that is syntactically valid but semantically incomplete.
### Hosted shell and code interpreter failure shields
When probing hosted tools through the Responses API, eliminate common setup ambiguity first:
- Force the tool path you want to test with the matching `tool_choice`. A text-only completion without forced tool choice is not a reliable negative result.
- Treat `container_auto` and `container_reference` differently. Use `container_auto` when the probe needs fresh container provisioning or skill attachment, and use `container_reference` only to reuse existing container state.
- Do not assume every environment field is accepted on every container mode. If the probe is about skills, validate that the chosen container mode actually supports skill attachment before treating an API error as a runtime defect.
- Check model-specific option support before chasing unrelated failures. Unsupported reasoning or model settings can invalidate the probe before the tool path is exercised.
- For hosted package installation, treat network-dependent setup as best-effort and separate install failures from the underlying tool behavior you are trying to observe.
- For prompt cache investigations, keep model, instructions, tool configuration, and cache key effectively identical across repeated runs before interpreting `cached_tokens`.
### Streaming behavior
Use when the uncertainty involves:
- Event ordering.
- Partial text delivery.
- Termination after interruption.
- Tool-call events in streams.
Probe suggestions:
- Normal streamed completion.
- Early local cancellation.
- Network interruption if it can be reproduced safely.
## What to Capture
For OpenAI probes, try to record:
- Request options that materially affect behavior.
- Response item types and their order.
- Whether fields are absent, null, empty, or transformed.
- Server status and error payload details for failures.
- Retry and backoff hints when present.
- Stable identifiers that help compare repeated runs, such as request IDs, response IDs, tool call IDs, or container IDs when available.
- Which environment-variable names were approved for the probe when live credentials were required.
Do not spend time rediscovering static documentation unless the runtime result seems to contradict what you expected. The value of this skill is in the observed behavior.
.agents/skills/runtime-behavior-probe/references/reporting-format.md
+118
View File
@@ -0,0 +1,118 @@
# Reporting Format
Lead with findings, not process. The user asked for investigation results, so the answer should start with the most important observed behaviors. Put the real news first.
## Recommended Order
1. Findings.
2. Validation approach.
3. Case matrix or condensed case summary.
4. Artifact status and brief run summary.
5. Optional implementation note.
## Findings Section
Make each finding answer one user-relevant question. Good findings usually include:
- What was observed.
- Why it matters.
- The condition under which it happens.
- What was held constant when the finding comes from a comparison probe.
- `scope`: The boundary of the finding, such as commit, model, Python version, live vs local, or repeat mode.
- `confidence`: `high`, `medium`, or `low`.
Avoid burying the main result under setup details.
Put `unexpected` or `negative` findings first. If there were no unexpected or negative findings in the executed cases, say that explicitly before the rest of the findings section.
If the probe was comparative, say whether the result supports:
- Pattern parity only.
- A broader quality claim.
Do not imply a broader quality equivalence than the executed cases justify.
## Validation Approach Section
Summarize:
- The runtime surface you exercised.
- The shape of the probe code, in overview only.
- Which categories of cases you covered.
- Which execution modes you used, including repeat counts or warm-up handling when relevant.
- Whether live credentials or external services were used.
- Any important state controls such as fresh state, cache reuse, cache busting, unique IDs, or cleanup verification.
- For comparison probes, what was held constant, what was varied, and whether output-shape or usage differences could still influence the conclusion.
- Whether the usual docs path or an official-docs fallback was used for contract-sensitive checks.
Keep this concise. The user needs enough detail to trust the result, not a line-by-line replay of the script.
## Case Summary
Include either the full matrix or a condensed summary. At minimum, show:
- Which scenarios were executed.
- Whether the run was a quick pilot, an expanded matrix, or both.
- Which ones produced `unexpected` or `negative` results.
- Which ones passed or failed when a real comparison basis existed.
- Which cases were blocked.
- Where the supporting evidence lived, or that it was deleted.
If the matrix is large, show the highest-value cases in the main response and keep the rest as a compact appendix or note.
## Artifact Status And Brief Run Summary
State one of these explicitly:
- Temporary artifacts were kept until the final response was drafted, then deleted after validation.
- Temporary artifacts were kept at `<path>` because the user asked to keep them.
- Temporary artifacts were kept at `<path>` because they are needed for follow-up analysis.
Even if artifacts were deleted, retain a short run summary such as:
- Probe command or runner shape.
- Runtime context summary such as commit, Python executable, Python version, or model.
- Artifact path and final status.
For benchmark or repeat-heavy probes, keeping artifacts for follow-up is often the right default even when the immediate report is done.
## Optional Implementation Note
Include this only when one clear defect was isolated and a short implementation hypothesis or minimal repro direction would help. Keep it brief. Do not turn the report into a broader next-step plan unless the user asked for that.
## Compact Template
Use this outline when you need a fast structure:
Findings:
- <finding 1>
held constant: <prompt/tool/state settings kept the same, if comparative>
scope: <commit/model/python/live-local/repeat-mode>
confidence: <high|medium|low>
- <finding 2>
held constant: <prompt/tool/state settings kept the same, if comparative>
scope: <commit/model/python/live-local/repeat-mode>
confidence: <high|medium|low>
Validation approach:
- Surface: <what was exercised>
- Probe code: <brief overview>
- Coverage: <success, edge, error, repeat-sensitive, and quality categories>
- Execution modes: <single-shot|repeat-N|warm-up + repeat-N>
- Comparison parity: <what was held constant and what varied, if comparative>
- Docs source: <MCP or official-docs fallback, if relevant>
Case summary:
| case_id | scenario | result_flag | status | note |
| --- | --- | --- | --- | --- |
| S1 | ... | expected | pass | ... |
| E1 | ... | negative | fail | ... |
Artifact status and brief run summary:
- Temporary artifacts were kept until the final response was drafted, then deleted.
- Summary: <command/runtime-context/artifact-status summary>
Optional implementation note:
- <brief hypothesis or minimal repro direction>
Adjust the format to the task, but preserve the ordering.
.agents/skills/runtime-behavior-probe/references/validation-matrix.md
+137
View File
@@ -0,0 +1,137 @@
# Validation Matrix
Use the matrix to decide what to probe before writing scripts. The goal is not exhaustive combinatorics; the goal is high-value coverage that is visible, explainable, and likely to reveal runtime surprises. The matrix should make the real news easy to scan.
## Minimum Columns
Use these columns unless the task clearly needs more:
- `case_id`: Stable identifier such as `S1`, `E3`, or `R2`.
- `scenario`: Short description of the behavior under test.
- `mode`: `single-shot`, `repeat-N`, or `warm-up + repeat-N`.
- `question`: The concrete runtime uncertainty this case is answering.
- `setup`: Inputs, environment, or preconditions required for the case.
- `observation_summary`: A compact summary of what actually happened.
- `result_flag`: `unexpected`, `negative`, `expected`, or `blocked`.
- `evidence`: Path, log reference, or `deleted`.
Add these columns when they materially improve the investigation:
- `comparison_basis`: The baseline, docs, or prior behavior you are comparing against.
- `variable_under_test`: The single factor that is intentionally changing in a comparison case.
- `held_constant`: Prompt shape, tool setup, model settings, or state rules that were intentionally kept the same.
- `output_constraint`: Any schema, length, or response-shape constraint used to keep the comparison fair.
- `status`: Use `pass`, `fail`, `unexpected-pass`, `unexpected-fail`, or `blocked` only when there is a credible comparison basis or control.
- `confidence`: `high`, `medium`, or `low`.
- `state_setup`: Fresh or reused state, cache strategy, unique IDs, and cleanup checks.
- `repeats`: Number of measured runs.
- `warm_up`: Whether a warm-up run was used and why.
- `variance`: Any useful spread or instability note across repeated runs.
- `usage_note`: Token, usage, or output-length note when it materially affects interpretation.
- `control`: Known-good comparison point for regression or behavior-change questions.
- `risk_profile`: `read-only`, `mutating`, or `costly` for live probes.
- `env_vars`: Exact environment-variable names the case plans to read.
- `approval`: `not-needed`, `pending`, or `approved` for cases that need user permission before execution.
Use `result_flag` as the fast scan field. It is what makes unexpected or negative findings jump out before the reader studies the full report.
Use `status` only when you have a real comparison basis. If the case is exploratory and there is no trustworthy baseline, prefer a strong `observation_summary` plus `result_flag` and `confidence` instead of pretending the result is a clean pass or fail.
## Choosing Execution Mode
Pick an execution mode before you run the case:
- Use `single-shot` for deterministic, one-run checks.
- Use `repeat-N` automatically when the question involves cache behavior, retries, streaming, interruptions, rate limiting, concurrency, or other run-to-run-sensitive behavior.
- Use `warm-up + repeat-N` when the first run is likely to include cold-start effects such as container provisioning, import caches, or prompt-cache population.
Use these defaults unless the task clearly needs something else:
- `repeat-3` for a quick screen of a repeat-sensitive question.
- `warm-up + repeat-10` for decision-grade latency comparisons or release-facing recommendations.
- For costly live probes, start at `repeat-3` and expand only if the answer is still unclear.
If it is genuinely unclear whether extra runs are worth the time or cost, ask the user before expanding the probe.
## Phase The Matrix
When the question is comparative or benchmark-like, do not jump straight to the largest matrix.
Start with a pilot:
- One control.
- One or two highest-signal success cases.
- The smallest repeat count that can disqualify a weak candidate quickly.
Expand only when:
- The candidate survives the pilot.
- The results are close enough that more samples matter.
- A major runtime surface is still uncovered.
- The user explicitly wants decision-grade evidence.
## Coverage Categories
Try to cover at least one case from each relevant category:
- `success`: Normal behavior that should work.
- `control`: Known-good comparison such as `origin/main`, the latest release, or the same request without the suspected option.
- `boundary`: Size, count, or parameter limits near a plausible edge.
- `invalid`: Bad inputs or unsupported combinations.
- `misconfig`: Missing key, wrong endpoint, bad permissions, or incompatible local setup.
- `transient`: Timeout, temporary server issue, network breakage, or rate limiting.
- `recovery`: Retry behavior, partial completion, duplicate submission, or cleanup.
- `concurrency`: Overlapping operations when shared state, ordering, or isolation may matter.
- `quality`: A harder or more open-ended sample when the user is asking about model intelligence, not just workflow parity.
If time is limited, prioritize categories in this order:
1. Known-good control when the question implies regression or drift.
2. Highest-risk success case.
3. Most plausible user-facing failure.
4. Most likely edge case with ambiguous behavior.
5. Cleanup or retry semantics.
6. Lower-probability extremes.
## Matrix Template
Use this compact template:
| case_id | scenario | mode | question | setup | state_setup | variable_under_test | held_constant | comparison_basis | observation_summary | result_flag | status | evidence |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| K1 | Known-good control | single-shot | Does the baseline still show the expected behavior? | Same probe against baseline target | Fresh state | none | current probe shape | `origin/main` or latest release | pending | pending | pending | pending |
| S1 | Baseline success | single-shot | What does the normal success path look like at runtime? | Valid config and representative input | Fresh state | none | representative input and setup | current docs or local expectation | pending | pending | pending | pending |
| R1 | Cache or retry behavior | warm-up + repeat-N | Does behavior change after the first run or across retries? | Same request repeated under controlled settings | Cache key or retry setup recorded | reuse versus fresh state | prompt shape and tool setup | same request without reuse, or docs if available | pending | pending | pending | pending |
| C1 | Model comparison pilot | warm-up + repeat-N | Does candidate B preserve the covered behavior while improving latency? | Same scenario across two models | Fresh state and stable IDs | model name | prompt shape, tool choice, and model settings parity | control model in the same probe | pending | pending | pending | pending |
| E1 | Invalid input | single-shot | How does the runtime reject a realistic bad input? | Missing required field | Fresh state | invalid field value | same request with valid field | same request with valid field | pending | pending | pending | pending |
| X1 | Concurrent overlap | repeat-N | Do overlapping runs interfere with each other? | Two or more overlapping operations | Unique IDs plus cleanup verification | overlap timing | same logical input | same request serialized, if available | pending | pending | pending | pending |
## Recording Results
Keep `question` unchanged after execution. Put the actual behavior in `observation_summary`, then mark the scan-friendly `result_flag`.
Use these `result_flag` values consistently:
- `unexpected`: The result diverged from the best current understanding in a surprising way.
- `negative`: The result exposed a user-relevant failure, risk, or sharp edge.
- `expected`: The result matched the current understanding and did not reveal new risk.
- `blocked`: The case did not produce a trustworthy observation.
Only fill `status` when there is a credible comparison basis. Otherwise use `observation_summary`, `result_flag`, and `confidence` to communicate what was learned without over-claiming certainty.
For comparison cases, use `observation_summary` and the final report to say whether the evidence supports pattern parity only or a broader quality claim.
If a case reveals a new branch of behavior, add a follow-up case instead of overloading the original one.
## Evidence Discipline
Treat a case as incomplete when:
- The observed output omits the key result you were testing.
- The script mixed multiple questions and the result is ambiguous.
- Hidden state, cache behavior, or previous runs may have influenced the result and were not controlled or documented.
- The question is whether behavior changed, but the case has no credible control or baseline to compare against.
- The case plans to read environment variables, but the exact variable names were not approved by the user before execution.
- The case was repeat-sensitive, but it ran only once without a clear rationale.
When this happens, narrow the probe and rerun. A smaller script with a cleaner result is better than a more complicated script that is hard to trust.
.agents/skills/runtime-behavior-probe/templates/python_probe.py
+227
View File
@@ -0,0 +1,227 @@
"""Disposable Python probe scaffold.
Copy this file to a temporary location and adapt it for one narrow question.
Recommended usage from the repository root:
uv run python /tmp/probe.py
If you want structured artifacts for repeat-heavy or benchmark probes:
PROBE_OUTPUT_DIR=/tmp/probe-run uv run python /tmp/probe.py
"""
from __future__ import annotations
import json
import os
import platform
import shutil
import statistics
import subprocess
import sys
import time
import uuid
from collections import Counter, defaultdict
from importlib import metadata
from pathlib import Path
SCENARIO = "replace-me"
RUN_LABEL = "replace-me"
MODE = "single-shot"
APPROVED_ENV_VARS: list[str] = []
OUTPUT_DIR_ENV = "PROBE_OUTPUT_DIR"
RESULTS: list[dict[str, object]] = []
def _git_value(*args: str) -> str:
result = subprocess.run(
["git", *args],
check=False,
capture_output=True,
text=True,
)
if result.returncode != 0:
return "unknown"
return result.stdout.strip() or "unknown"
def _package_version(name: str) -> str | None:
try:
return metadata.version(name)
except metadata.PackageNotFoundError:
return None
def _output_dir() -> Path | None:
value = os.getenv(OUTPUT_DIR_ENV)
if not value:
return None
return Path(value)
def _write_json(path: Path, payload: object) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
def emit(kind: str, **payload: object) -> None:
print(
json.dumps(
{
"ts": round(time.time(), 3),
"kind": kind,
**payload,
},
sort_keys=True,
)
)
def runtime_context() -> dict[str, object]:
approved = {name: ("set" if os.getenv(name) else "unset") for name in APPROVED_ENV_VARS}
package_versions = {
name: version
for name in ("openai", "agents")
if (version := _package_version(name)) is not None
}
return {
"scenario": SCENARIO,
"run_label": RUN_LABEL,
"mode": MODE,
"cwd": os.getcwd(),
"script_path": str(Path(__file__).resolve()),
"python_executable": sys.executable,
"python_version": sys.version.split()[0],
"platform": platform.platform(),
"git_commit": _git_value("rev-parse", "HEAD"),
"git_branch": _git_value("rev-parse", "--abbrev-ref", "HEAD"),
"uv_path": shutil.which("uv"),
"package_versions": package_versions,
"approved_env_vars": approved,
"output_dir": str(_output_dir()) if _output_dir() else None,
}
def start_case(case_id: str, *, mode: str = MODE, note: str | None = None) -> None:
emit("case_start", case_id=case_id, mode=mode, note=note)
def record_case_result(
case_id: str,
observation_summary: str,
result_flag: str,
*,
mode: str = MODE,
is_warmup: bool = False,
total_latency_s: float | None = None,
first_token_latency_s: float | None = None,
metrics: dict[str, object] | None = None,
error: str | None = None,
) -> None:
payload: dict[str, object] = {
"case_id": case_id,
"mode": mode,
"is_warmup": is_warmup,
"observation_summary": observation_summary,
"result_flag": result_flag,
"metrics": metrics or {},
"error": error,
}
if total_latency_s is not None:
payload["total_latency_s"] = total_latency_s
if first_token_latency_s is not None:
payload["first_token_latency_s"] = first_token_latency_s
RESULTS.append(payload)
emit("case_result", **payload)
def summarize_results() -> dict[str, object]:
by_case: defaultdict[str, list[dict[str, object]]] = defaultdict(list)
for result in RESULTS:
by_case[str(result["case_id"])].append(result)
summary_cases: dict[str, object] = {}
for case_id, items in by_case.items():
measured = [item for item in items if not bool(item.get("is_warmup"))]
latencies = [
float(item["total_latency_s"])
for item in measured
if item.get("total_latency_s") is not None
]
first_token_latencies = [
float(item["first_token_latency_s"])
for item in measured
if item.get("first_token_latency_s") is not None
]
result_flags = Counter(str(item["result_flag"]) for item in measured or items)
observations = [str(item["observation_summary"]) for item in (measured or items)[:3]]
summary_cases[case_id] = {
"mode": str(items[-1]["mode"]),
"runs": len(measured),
"warmups": len(items) - len(measured),
"result_flags": dict(result_flags),
"median_total_latency_s": (statistics.median(latencies) if latencies else None),
"mean_total_latency_s": statistics.mean(latencies) if latencies else None,
"median_first_token_latency_s": (
statistics.median(first_token_latencies) if first_token_latencies else None
),
"observations": observations,
}
return {
"scenario": SCENARIO,
"run_label": RUN_LABEL,
"mode": MODE,
"result_count": len(RESULTS),
"cases": summary_cases,
"result_flags": dict(Counter(str(item["result_flag"]) for item in RESULTS)),
}
def finalize(exit_code: int) -> None:
metadata_payload = {
"exit_code": exit_code,
"runtime_context": runtime_context(),
}
summary_payload = summarize_results()
emit("summary", metadata=metadata_payload, summary=summary_payload)
output_dir = _output_dir()
if not output_dir:
return
metadata_path = output_dir / "metadata.json"
results_path = output_dir / "results.json"
summary_path = output_dir / "summary.json"
_write_json(metadata_path, metadata_payload)
_write_json(results_path, RESULTS)
_write_json(summary_path, summary_payload)
emit(
"artifact_paths",
metadata_path=str(metadata_path),
results_path=str(results_path),
summary_path=str(summary_path),
)
def main() -> int:
case_id = os.getenv("PROBE_CASE_ID", f"case-{uuid.uuid4().hex[:8]}")
emit("banner", context=runtime_context())
start_case(case_id)
# Replace this block with the narrow runtime question you want to test.
observation = "replace-me"
result_flag = "expected"
record_case_result(
case_id=case_id,
observation_summary=observation,
result_flag=result_flag,
)
finalize(exit_code=0)
return 0
if __name__ == "__main__":
raise SystemExit(main())