wehub-resource-sync
138 lines
8.8 KiB
Markdown
138 lines
8.8 KiB
Markdown
# Validation Matrix
|
|
|
|
Use the matrix to decide what to probe before writing scripts. The goal is not exhaustive combinatorics; the goal is high-value coverage that is visible, explainable, and likely to reveal runtime surprises. The matrix should make the real news easy to scan.
|
|
|
|
## Minimum Columns
|
|
|
|
Use these columns unless the task clearly needs more:
|
|
|
|
- `case_id`: Stable identifier such as `S1`, `E3`, or `R2`.
|
|
- `scenario`: Short description of the behavior under test.
|
|
- `mode`: `single-shot`, `repeat-N`, or `warm-up + repeat-N`.
|
|
- `question`: The concrete runtime uncertainty this case is answering.
|
|
- `setup`: Inputs, environment, or preconditions required for the case.
|
|
- `observation_summary`: A compact summary of what actually happened.
|
|
- `result_flag`: `unexpected`, `negative`, `expected`, or `blocked`.
|
|
- `evidence`: Path, log reference, or `deleted`.
|
|
|
|
Add these columns when they materially improve the investigation:
|
|
|
|
- `comparison_basis`: The baseline, docs, or prior behavior you are comparing against.
|
|
- `variable_under_test`: The single factor that is intentionally changing in a comparison case.
|
|
- `held_constant`: Prompt shape, tool setup, model settings, or state rules that were intentionally kept the same.
|
|
- `output_constraint`: Any schema, length, or response-shape constraint used to keep the comparison fair.
|
|
- `status`: Use `pass`, `fail`, `unexpected-pass`, `unexpected-fail`, or `blocked` only when there is a credible comparison basis or control.
|
|
- `confidence`: `high`, `medium`, or `low`.
|
|
- `state_setup`: Fresh or reused state, cache strategy, unique IDs, and cleanup checks.
|
|
- `repeats`: Number of measured runs.
|
|
- `warm_up`: Whether a warm-up run was used and why.
|
|
- `variance`: Any useful spread or instability note across repeated runs.
|
|
- `usage_note`: Token, usage, or output-length note when it materially affects interpretation.
|
|
- `control`: Known-good comparison point for regression or behavior-change questions.
|
|
- `risk_profile`: `read-only`, `mutating`, or `costly` for live probes.
|
|
- `env_vars`: Exact environment-variable names the case plans to read.
|
|
- `approval`: `not-needed`, `pending`, or `approved` for cases that need user permission before execution.
|
|
|
|
Use `result_flag` as the fast scan field. It is what makes unexpected or negative findings jump out before the reader studies the full report.
|
|
|
|
Use `status` only when you have a real comparison basis. If the case is exploratory and there is no trustworthy baseline, prefer a strong `observation_summary` plus `result_flag` and `confidence` instead of pretending the result is a clean pass or fail.
|
|
|
|
## Choosing Execution Mode
|
|
|
|
Pick an execution mode before you run the case:
|
|
|
|
- Use `single-shot` for deterministic, one-run checks.
|
|
- Use `repeat-N` automatically when the question involves cache behavior, retries, streaming, interruptions, rate limiting, concurrency, or other run-to-run-sensitive behavior.
|
|
- Use `warm-up + repeat-N` when the first run is likely to include cold-start effects such as container provisioning, import caches, or prompt-cache population.
|
|
|
|
Use these defaults unless the task clearly needs something else:
|
|
|
|
- `repeat-3` for a quick screen of a repeat-sensitive question.
|
|
- `warm-up + repeat-10` for decision-grade latency comparisons or release-facing recommendations.
|
|
- For costly live probes, start at `repeat-3` and expand only if the answer is still unclear.
|
|
|
|
If it is genuinely unclear whether extra runs are worth the time or cost, ask the user before expanding the probe.
|
|
|
|
## Phase The Matrix
|
|
|
|
When the question is comparative or benchmark-like, do not jump straight to the largest matrix.
|
|
|
|
Start with a pilot:
|
|
|
|
- One control.
|
|
- One or two highest-signal success cases.
|
|
- The smallest repeat count that can disqualify a weak candidate quickly.
|
|
|
|
Expand only when:
|
|
|
|
- The candidate survives the pilot.
|
|
- The results are close enough that more samples matter.
|
|
- A major runtime surface is still uncovered.
|
|
- The user explicitly wants decision-grade evidence.
|
|
|
|
## Coverage Categories
|
|
|
|
Try to cover at least one case from each relevant category:
|
|
|
|
- `success`: Normal behavior that should work.
|
|
- `control`: Known-good comparison such as `origin/main`, the latest release, or the same request without the suspected option.
|
|
- `boundary`: Size, count, or parameter limits near a plausible edge.
|
|
- `invalid`: Bad inputs or unsupported combinations.
|
|
- `misconfig`: Missing key, wrong endpoint, bad permissions, or incompatible local setup.
|
|
- `transient`: Timeout, temporary server issue, network breakage, or rate limiting.
|
|
- `recovery`: Retry behavior, partial completion, duplicate submission, or cleanup.
|
|
- `concurrency`: Overlapping operations when shared state, ordering, or isolation may matter.
|
|
- `quality`: A harder or more open-ended sample when the user is asking about model intelligence, not just workflow parity.
|
|
|
|
If time is limited, prioritize categories in this order:
|
|
|
|
1. Known-good control when the question implies regression or drift.
|
|
2. Highest-risk success case.
|
|
3. Most plausible user-facing failure.
|
|
4. Most likely edge case with ambiguous behavior.
|
|
5. Cleanup or retry semantics.
|
|
6. Lower-probability extremes.
|
|
|
|
## Matrix Template
|
|
|
|
Use this compact template:
|
|
|
|
| case_id | scenario | mode | question | setup | state_setup | variable_under_test | held_constant | comparison_basis | observation_summary | result_flag | status | evidence |
|
|
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|
|
| K1 | Known-good control | single-shot | Does the baseline still show the expected behavior? | Same probe against baseline target | Fresh state | none | current probe shape | `origin/main` or latest release | pending | pending | pending | pending |
|
|
| S1 | Baseline success | single-shot | What does the normal success path look like at runtime? | Valid config and representative input | Fresh state | none | representative input and setup | current docs or local expectation | pending | pending | pending | pending |
|
|
| R1 | Cache or retry behavior | warm-up + repeat-N | Does behavior change after the first run or across retries? | Same request repeated under controlled settings | Cache key or retry setup recorded | reuse versus fresh state | prompt shape and tool setup | same request without reuse, or docs if available | pending | pending | pending | pending |
|
|
| C1 | Model comparison pilot | warm-up + repeat-N | Does candidate B preserve the covered behavior while improving latency? | Same scenario across two models | Fresh state and stable IDs | model name | prompt shape, tool choice, and model settings parity | control model in the same probe | pending | pending | pending | pending |
|
|
| E1 | Invalid input | single-shot | How does the runtime reject a realistic bad input? | Missing required field | Fresh state | invalid field value | same request with valid field | same request with valid field | pending | pending | pending | pending |
|
|
| X1 | Concurrent overlap | repeat-N | Do overlapping runs interfere with each other? | Two or more overlapping operations | Unique IDs plus cleanup verification | overlap timing | same logical input | same request serialized, if available | pending | pending | pending | pending |
|
|
|
|
## Recording Results
|
|
|
|
Keep `question` unchanged after execution. Put the actual behavior in `observation_summary`, then mark the scan-friendly `result_flag`.
|
|
|
|
Use these `result_flag` values consistently:
|
|
|
|
- `unexpected`: The result diverged from the best current understanding in a surprising way.
|
|
- `negative`: The result exposed a user-relevant failure, risk, or sharp edge.
|
|
- `expected`: The result matched the current understanding and did not reveal new risk.
|
|
- `blocked`: The case did not produce a trustworthy observation.
|
|
|
|
Only fill `status` when there is a credible comparison basis. Otherwise use `observation_summary`, `result_flag`, and `confidence` to communicate what was learned without over-claiming certainty.
|
|
|
|
For comparison cases, use `observation_summary` and the final report to say whether the evidence supports pattern parity only or a broader quality claim.
|
|
|
|
If a case reveals a new branch of behavior, add a follow-up case instead of overloading the original one.
|
|
|
|
## Evidence Discipline
|
|
|
|
Treat a case as incomplete when:
|
|
|
|
- The observed output omits the key result you were testing.
|
|
- The script mixed multiple questions and the result is ambiguous.
|
|
- Hidden state, cache behavior, or previous runs may have influenced the result and were not controlled or documented.
|
|
- The question is whether behavior changed, but the case has no credible control or baseline to compare against.
|
|
- The case plans to read environment variables, but the exact variable names were not approved by the user before execution.
|
|
- The case was repeat-sensitive, but it ran only once without a clear rationale.
|
|
|
|
When this happens, narrow the probe and rerun. A smaller script with a cleaner result is better than a more complicated script that is hard to trust.
|