chore: import upstream snapshot with attribution
This commit is contained in:
@@ -0,0 +1,63 @@
|
||||
# Runtime experiment toggles ("levers")
|
||||
|
||||
OpenSquilla exposes a family of opt-in runtime behaviors that scripted or
|
||||
harness-controlled runs can enable per run without changing code or config
|
||||
files. This page records the conventions those toggles must follow and how a
|
||||
calling harness can verify that a toggle it requested was actually delivered.
|
||||
The companion tooling lives in `scripts/experiments/`.
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Naming**: `OPENSQUILLA_<AREA>_<KNOB>` (e.g.
|
||||
`OPENSQUILLA_PROVIDER_HISTORY_DEDUP`,
|
||||
`OPENSQUILLA_TOOL_REPEAT_NUDGE_THRESHOLD`).
|
||||
- **Single parse site**: environment values are parsed only in
|
||||
`engine/turn_runner/agent_bootstrap_stage.py` (`_*_from_env` helpers), flow
|
||||
into `AgentConfig` fields, and are consumed by runtime resolvers. Nothing
|
||||
else in the engine reads these variables directly.
|
||||
- **Default off**: with no toggle set, the runtime behaves like stock
|
||||
OpenSquilla. A toggle must never change behavior for users who have not set
|
||||
it.
|
||||
- **Strict values**: unrecognized values raise instead of being silently
|
||||
ignored, so a run manifest cannot record an override the run did not
|
||||
actually apply.
|
||||
- **Provider differences live in policy tables**: per-provider behavior
|
||||
belongs in `OpenAICompatPolicy`, `ProviderContextProfile`, or
|
||||
`reasoning_dialects` fields — never in model-name conditionals at call
|
||||
sites.
|
||||
|
||||
## Delivery verification
|
||||
|
||||
A harness that sets toggles can verify delivery end to end:
|
||||
|
||||
1. The harness allowlist decides which variables are passed into the
|
||||
container (`docker exec -e ...`).
|
||||
2. The harness echoes the delivered environment into the run's
|
||||
`metadata.json` under `agent.controls.progress_watchdog_env`. This echo is
|
||||
written by the harness adapter unconditionally — it does not depend on any
|
||||
runtime toggle, including the watchdog mode itself.
|
||||
3. `scripts/experiments/exp_finalize.py` gates a finished run on the expected
|
||||
environment (`AGENT_ENV_DELIVERY_VARS`), and
|
||||
`scripts/experiments/check_treatment_delivery.py` asserts the resulting
|
||||
provider payload shape (e.g. expected proof budget, reasoning effort, and
|
||||
a bounded number of reasoning fallbacks).
|
||||
|
||||
## Adding a new toggle
|
||||
|
||||
A new toggle must:
|
||||
|
||||
1. be added to the calling harness's allowlist,
|
||||
2. default off, and
|
||||
3. be added to `AGENT_ENV_DELIVERY_VARS` if it can affect task outcomes, so
|
||||
the delivery gate covers it.
|
||||
|
||||
## Reproducing older behavior
|
||||
|
||||
Two defaults were flipped to off when this code was merged; runs that want
|
||||
the previous behavior should pin them explicitly:
|
||||
|
||||
- `OPENSQUILLA_PROGRESS_WATCHDOG_MODE=warn_model` (merged default: `off`;
|
||||
other values: `log`, `block`). Harness-controlled runs should always pin
|
||||
this mode explicitly rather than relying on the default.
|
||||
- `OPENSQUILLA_TOOL_REPEAT_NUDGE_THRESHOLD=3` (merged default: `0`,
|
||||
disabled).
|
||||
Reference in New Issue
Block a user