wehub-resource-sync
737 lines
32 KiB
Markdown
737 lines
32 KiB
Markdown
# bench_env task code spec
|
||
|
||
> This is the **hard spec** for all Python code under `bench_env/task/` — naming, forbidden patterns, error handling, file responsibilities, docstrings. Every new or modified task must comply; PR reviews follow this doc.
|
||
>
|
||
> Authoring workflow: [`TASK_AUTHORING_GUIDE.md`](TASK_AUTHORING_GUIDE.md); testing: [`TASK_TESTING_GUIDE.md`](TASK_TESTING_GUIDE.md).
|
||
|
||
---
|
||
|
||
## 1. File responsibilities
|
||
|
||
Per-suite file responsibilities:
|
||
|
||
| File | Responsibility | Must NOT contain |
|
||
|---|---|---|
|
||
| `tasks.py` / `defs/<Name>.py` | Task class definitions | Data-access helpers, generic utilities, App-private constants, custom base classes |
|
||
| `app.py` | App accessor (subclass of `BaseApp`): data methods / answer methods / `check_*` methods / `prepare_state_with_*` helpers / App-private samplers / `expected_changes` constants | Task-specific decision logic, business rules, runtime-environment operations |
|
||
| `__init__.py` | Module marker | Any real code |
|
||
|
||
### 1.1 Task files (`tasks.py` / `defs/*.py`)
|
||
|
||
**Allowed**:
|
||
|
||
- Task class definitions (inheriting from `BaseTask` / `CriteriaTask` / `AnswerTask`)
|
||
- Importing and composing `expected_changes` constants from app.py
|
||
- import statements
|
||
- Task Index comments (only in `tasks.py`)
|
||
|
||
**Forbidden**:
|
||
|
||
- Custom base classes, mixins, abstract intermediaries (all tasks must inherit a standard base class directly)
|
||
- Module-level helper functions (`_chk()` / `_build_xxx()`) → belong in app.py or utils.py
|
||
- App-private data constants (sampling pools, route tables, value-mapping constants) → belong in app.py
|
||
- Inter-task inheritance (task A must not inherit from task B then tweak a single check; it looks like dedup but actually couples them)
|
||
- Private compute/aggregation methods on the task class (`_rain_days_next_week()` etc.) → generalize into App methods
|
||
- Sampler function definitions → `sample_*` belongs on the App class
|
||
|
||
**Authoring principles**:
|
||
|
||
1. **Each task is self-contained** — does not inherit from another task, does not depend on file-level shared helpers
|
||
2. **Inherit only from the standard base classes**
|
||
3. **Declarative first** — if you can express it via `answer = ".path"` / `criteria = {...}`, don't write a method
|
||
4. **Task class only composes; it doesn't compute** — data traversal, aggregation, and computation move to App methods
|
||
|
||
### 1.2 App class (`app.py`)
|
||
|
||
**Encapsulate**:
|
||
|
||
| Yes | No |
|
||
|---|---|
|
||
| Data-access complexity (multi-step lookup, fuzzy matching, field compatibility) | Task-specific conditional branching |
|
||
| Data traversal, aggregation (even if only one task uses it today) | Embedded UI-layer formatting |
|
||
| init vs current diffs (generic comparison) | Hard-coded business rules (`"转账" in name`) |
|
||
| Schema-coupled setup (`prepare_state_with_*`) | Direct manipulation of `env.get_state()` / `env.set_state()` |
|
||
| App-private samplers (`@staticmethod sample_*`) | UI layout logic |
|
||
|
||
**Decision criteria**:
|
||
|
||
1. About where data lives, how to fetch it, what fields exist → App
|
||
2. About what to do with data after fetching → Task
|
||
3. Hard-codes a specific business rule → Task
|
||
4. Used by only one task? **Still check whether it's data-access complexity** — if so, App; if not, Task
|
||
|
||
For the detailed three-tier breakdown (data / answer / check methods) see [`TASK_AUTHORING_GUIDE.md`](TASK_AUTHORING_GUIDE.md) §2.
|
||
|
||
### 1.3 `task/utils.py`
|
||
|
||
Cross-suite pure functions: text processing, time utilities, data parsing, check combinators (`check_alternatives`).
|
||
|
||
`check_alternatives(*check_arrays)` is candidate-wise OR: same-index checks across arrays form one candidate, the first all-pass candidate is returned, and if none pass the first candidate is returned for diagnostics. Arrays must be non-empty and the same length.
|
||
|
||
**Forbidden**:
|
||
|
||
- Defining shared utilities locally in a task file
|
||
- Inlining generic parsing logic in app.py
|
||
|
||
---
|
||
|
||
## 2. Naming conventions
|
||
|
||
### 2.1 App class names
|
||
|
||
Correspond to `manifest.id`, PascalCase, **without** an `App` suffix:
|
||
|
||
| `manifest.id` | Correct | Wrong |
|
||
|---|---|---|
|
||
| `wechat` | `Wechat` | `WechatApp` |
|
||
| `bilibili` | `Bilibili` | `BilibiliApp` |
|
||
| `railway12306` | `Railway12306` | `RailwayApp` |
|
||
|
||
### 2.2 Task class names
|
||
|
||
Must **accurately reflect the task goal**:
|
||
|
||
```python
|
||
# ❌ Name doesn't match behavior
|
||
class BalanceThresholdCheck(AnswerTask): # Actually just reads balance; no threshold
|
||
class ClearHistory(BaseTask): # Actually sets map orientation
|
||
|
||
# ✅
|
||
class CheckBalance(AnswerTask):
|
||
class SetMapOrientation(BaseTask):
|
||
```
|
||
|
||
### 2.3 `objective` must match content
|
||
|
||
```python
|
||
# ❌ objective is wrong
|
||
class OpenChatWithContact(CriteriaTask):
|
||
objective = "query" # Actually a navigation operation
|
||
|
||
# ✅
|
||
class OpenChatWithContact(CriteriaTask):
|
||
objective = "operate"
|
||
```
|
||
|
||
### 2.4 Objective and phrasing alignment
|
||
|
||
| Objective | Correct phrasing | Wrong pattern |
|
||
|---|---|---|
|
||
| `operate` | "Set XX to YY", "Open XX" | "Search XX, view YY" (viewing without output) |
|
||
| `query` | "View/check XX and tell me", "How much is XX?" | "Set XX" (state-changing instruction) |
|
||
| `hybrid` | "Find XX for me, tell me how many" | Same operate/query wrong patterns |
|
||
|
||
### 2.5 `expected_changes` constants
|
||
|
||
Named `<APP_NAME>_<ACTION>_CHANGES`, defined in the corresponding `app.py`:
|
||
|
||
```python
|
||
# wechat/app.py
|
||
WECHAT_SEND_CHANGES = ["wechat.chats"]
|
||
WECHAT_MOMENT_CHANGES = ["wechat.moments"]
|
||
```
|
||
|
||
---
|
||
|
||
## 3. Template authoring
|
||
|
||
### 3.1 Express intent, not steps
|
||
|
||
```python
|
||
# ❌ Step description
|
||
templates = ["搜索地点'{place}'，查看从当前位置到该地点的驾车路线"]
|
||
|
||
# ✅ Intent
|
||
templates = ["帮我在地图上找到从当前位置到'{place}'的驾车路线"]
|
||
```
|
||
|
||
### 3.2 Don't make the Agent "view" without output
|
||
|
||
If the instruction contains "view"/"check":
|
||
|
||
- Make it an `AnswerTask` and let the Agent reply → "View XX and tell me"
|
||
- Or rephrase to drop the "view" → "Open the XX page for me"
|
||
|
||
### 3.3 Templates must not leak answers
|
||
|
||
`AnswerTask` `templates` **must not contain placeholders for the answer** (e.g., `{phone}` / `{income}` / `{balance}`). The answer should be derived from App state via `get_answer()` or declarative `answer`. Declarative `answer` reads `input.apps_init` by default; use an explicit `get_answer()` with `input.apps` for final-state answers after an operation.
|
||
|
||
```python
|
||
# ❌ Template leaks the answer
|
||
templates = ["找到好友'{name}'，并记录其电话号码 {phone}"]
|
||
|
||
# ✅ Drop {phone}, use a declarative answer
|
||
templates = ["在支付宝里找到好友'{name}'，告诉我他的电话号码"]
|
||
answer = ".contacts[name={name}].phone"
|
||
```
|
||
|
||
### 3.4 `{param}` position inside templates
|
||
|
||
When a bool parameter combined with `values` dict is rendered, the full sentence must read naturally:
|
||
|
||
```python
|
||
# ✅ Renders as: "在 Spotify 中关闭'向他人展示我的收听活动'"
|
||
templates = ["在 Spotify 中{share_activity}'向他人展示我的收听活动'"]
|
||
parameters = {"share_activity": {"type": "bool",
|
||
"values": {"开启": True, "关闭": False}}}
|
||
|
||
# ❌ Renders as: "在隐私设置中关闭 XXX 开启并确认状态更新"
|
||
templates = ["在隐私设置中关闭 XXX {share_off} 并确认状态更新"]
|
||
```
|
||
|
||
### 3.5 L3 / L4 tasks should provide multiple template variants
|
||
|
||
To prevent the Agent from overfitting specific wording. **Critical constraint**: every variant must require the same answer content (same number of slots, same fields, same information).
|
||
|
||
```python
|
||
class FilterHeadphones(CriteriaTask):
|
||
templates = [
|
||
"帮我找最便宜的全新 Sony 耳机，只看日本发货且包邮的，告诉我有几款",
|
||
"我想要一副 Sony 的新耳机，从日本发货、不要运费的那种，有多少选择？",
|
||
]
|
||
```
|
||
|
||
---
|
||
|
||
## 4. Error handling: task-design error vs. agent execution failure
|
||
|
||
**One of the most important rules.**
|
||
|
||
| Situation | Correct | Wrong |
|
||
|---|---|---|
|
||
| Environment data missing (contact doesn't exist, DB empty, fixture missing) | App method `raise ValueError(...)` or task `raise RuntimeError(...)` | `return False` ❌ |
|
||
| App accessor data missing (field not found, no query result) | App method `raise ValueError(...)` | Return `""` / `None` ❌ |
|
||
| Agent failed to act (wrong route, wrong value) | `passed=False` | `raise RuntimeError(...)` ❌ |
|
||
|
||
```python
|
||
# ✅ App accessor raises when data is missing
|
||
class Map(BaseApp):
|
||
def place_address(self, name):
|
||
place = self._find_place(name)
|
||
if not place:
|
||
raise ValueError(f"Place {name!r} not found in state")
|
||
return place["address"]
|
||
|
||
# ❌ Silent empty return; task ends up with _require_runtime_answer() boilerplate
|
||
class Map(BaseApp):
|
||
def place_address(self, name):
|
||
place = self._find_place(name)
|
||
return place.get("address", "") if place else ""
|
||
```
|
||
|
||
**Why this matters**:
|
||
|
||
- `raise RuntimeError` → there's a bug in the task or environment; needs fixing
|
||
- `passed=False` → the Agent didn't succeed; a normal evaluation outcome
|
||
- Conflating the two attributes a sampler bug to the Agent silently
|
||
|
||
---
|
||
|
||
## 5. No defensive coding
|
||
|
||
**All code** under `bench_env/task/` must not defensively guard. Missing data, missing keys, type errors — they should surface as exceptions.
|
||
|
||
| ❌ Forbidden | ✅ Correct |
|
||
|---|---|
|
||
| `latest_order or {}` + `.get()` | Direct key access `latest_order["field"]` |
|
||
| `.get("key", "")` for a required field | `["key"]` |
|
||
| `(passenger or {}).get("name", "") or "Unknown"` | Declarative `answer = ".passengers[isDefault=True].name"` |
|
||
| `if x is not None` guard | Use `x` directly |
|
||
| `try / except` with a fallback return | Call directly and let the exception propagate |
|
||
| `input.apps.get("xxx", {})` | `input.apps["xxx"]` (framework guarantees a dict) |
|
||
| `(input.apps_init or {}).get("xxx")` | `input.apps_init["xxx"]` |
|
||
| `input.os or {}` | `input.os` |
|
||
|
||
**Legitimate handling of Agent failures**: use explicit ternaries in check dicts, not `(x or {}).get()`:
|
||
|
||
```python
|
||
# ✅ Agent may not have created the order; order being None is legitimate
|
||
return [{
|
||
"field": "newPendingOrder.trainNo",
|
||
"expected": target_train["trainNo"],
|
||
"actual": order["trainNo"] if order else None,
|
||
"passed": order is not None and order["trainNo"] == target_train["trainNo"],
|
||
}]
|
||
|
||
# ❌ or {} masks None
|
||
"actual": (order or {}).get("trainNo")
|
||
```
|
||
|
||
**No fallback returns in `get_answer()`**:
|
||
|
||
```python
|
||
# ❌
|
||
def get_answer(self, input):
|
||
temp = w.weather_now(self.p.city).get("temp")
|
||
return temp if temp is not None else "无法判断"
|
||
|
||
# ✅ Missing data should be raised by the App accessor
|
||
def get_answer(self, input):
|
||
return Weather(input.apps["weather"]).current_temp(self.p.city)
|
||
```
|
||
|
||
---
|
||
|
||
## 6. Forbidden time APIs
|
||
|
||
The following APIs **must not** be used under `bench_env/task/` to obtain or derive "current time":
|
||
|
||
| ❌ Forbidden | ✅ Replacement |
|
||
|---|---|
|
||
| `datetime.date.today()` | `sim_today(os_state)` |
|
||
| `datetime.datetime.now()` | `sim_datetime(os_state)` |
|
||
| `time.time()` | `now_ms(os_state)` |
|
||
| `datetime.datetime.fromtimestamp(time.time())` | `sim_datetime(os_state)` |
|
||
|
||
**No fallback to local time** — if `os_state` lacks `time`, `raise ValueError`; don't silently degrade.
|
||
|
||
**Exception**: converting an already-stored timestamp (e.g., `transferRecords[].timestamp`) with `datetime.datetime.fromtimestamp(ts)` is fine — this isn't about local time vs. simulated time, just formatting an existing absolute value.
|
||
|
||
---
|
||
|
||
## 7. CriteriaTask rules
|
||
|
||
### 7.1 `criteria` must be a class variable
|
||
|
||
```python
|
||
# ❌
|
||
class EnableDarkMode(CriteriaTask):
|
||
@property
|
||
def criteria(self):
|
||
return {"settings.general.darkMode": True}
|
||
|
||
# ✅
|
||
class EnableDarkMode(CriteriaTask):
|
||
criteria = {"settings.general.darkMode": True}
|
||
```
|
||
|
||
### 7.2 Parameterize with `"{param}"` template
|
||
|
||
```python
|
||
# ❌
|
||
@property
|
||
def criteria(self):
|
||
return {"searchForm.from": self.p.station}
|
||
|
||
# ✅
|
||
criteria = {"searchForm.from": "{station}"}
|
||
```
|
||
|
||
**The `"{param}"` template preserves the original Python type** for all parameter types — when the entire value is a pure single-parameter reference (e.g., `"{flag}"`, `"{count}"`), the framework returns `self.params[key]` directly, so `bool`/`int`/`float` are not converted to strings by `str.format()`. Mixed templates (e.g., `"prefix-{key}"`) still go through `str.format()`.
|
||
|
||
### 7.3 Value mapping uses `values` dict; no `_XXX_MAP`
|
||
|
||
```python
|
||
# ❌
|
||
_SIZE_MAP = {"最小": 0, "较小": 1, ...}
|
||
@property
|
||
def criteria(self):
|
||
return {"settings.fontSizeLevel": _SIZE_MAP[self.p.size_label]}
|
||
|
||
# ✅
|
||
parameters = {"font_size": {"type": "enum",
|
||
"values": {"最小": 0, "较小": 1, "标准": 2}}}
|
||
criteria = {"settings.fontSizeLevel": "{font_size}"}
|
||
```
|
||
|
||
### 7.4 Parameter semantics must match the store
|
||
|
||
```python
|
||
# ❌ Reversed semantics, forces a `not`
|
||
parameters = {"share_off": {"type": "bool", "default": True}}
|
||
@property
|
||
def criteria(self):
|
||
return {"settings.shareActivity": not self.p.share_off}
|
||
|
||
# ✅ Parameter value maps directly to store value
|
||
parameters = {"share_activity": {"type": "bool",
|
||
"values": {"开启": True, "关闭": False}}}
|
||
criteria = {"settings.shareActivity": "{share_activity}"}
|
||
```
|
||
|
||
---
|
||
|
||
## 8. `check_goals` return format
|
||
|
||
### 8.1 Each check must include `passed`
|
||
|
||
The framework raises `ValueError` for missing `passed` in `is_successful()` / `evaluate()`.
|
||
|
||
| Field | Type | Required | Description |
|
||
|---|---|---|---|
|
||
| `field` | `str` | yes | Check item name |
|
||
| `expected` | `Any` | yes | Expected value (must be readable) |
|
||
| `actual` | `Any` | yes | Actual value (must be readable) |
|
||
| `passed` | `bool` | **yes** | Pass / fail |
|
||
|
||
### 8.2 `expected` / `actual` must be diagnostic
|
||
|
||
These two values appear directly in logs and are the only clue when a task fails. **Do not write `expected=True, actual=None`**:
|
||
|
||
```python
|
||
# ❌ No diagnostic value
|
||
{"expected": True, "actual": order, ...}
|
||
# Log shows: expected=True, actual=None → can't tell what was expected or what the Agent did
|
||
|
||
# ✅ Human-readable summary
|
||
{"expected": "上海→南京 2026-03-21 G7002 二等 ×1 (赵宇轩)",
|
||
"actual": "未创建新订单", ...}
|
||
```
|
||
|
||
Full `check_goals` authoring rules: [`TASK_AUTHORING_GUIDE.md`](TASK_AUTHORING_GUIDE.md) §4.7.
|
||
|
||
---
|
||
|
||
## 9. Direct `JudgeInput` access
|
||
|
||
`JudgeInput.apps` / `apps_init` / `os` are guaranteed dicts by the framework — **index them directly**:
|
||
|
||
```python
|
||
# ✅
|
||
rail = Railway12306(
|
||
input.apps["railway12306"],
|
||
init=input.apps_init["railway12306"],
|
||
)
|
||
|
||
# ❌
|
||
app = Wechat(
|
||
input.apps.get("wechat", {}),
|
||
init=(input.apps_init or {}).get("wechat"),
|
||
)
|
||
```
|
||
|
||
A missing key means a config bug — let `KeyError` surface; don't mask it with `.get()` or `or {}`.
|
||
|
||
### 9.1 Declarative `answer` resolves against initial state
|
||
|
||
**All declarative `AnswerTask.answer` forms (path / tuple / dict / callable) read from `input.apps_init`** — the state captured at task setup. This matches pure-query semantics: the ground truth was frozen the moment the task started; whatever the Agent does shouldn't change "what the right answer was."
|
||
|
||
| Form | Reads from |
|
||
|---|---|
|
||
| `answer = ".x.y"` / `"app:.x.y"` | `input.apps_init` |
|
||
| `answer = (".x", fn)` | `input.apps_init` (then `fn`) |
|
||
| `answer = {"slot": ".x", ...}` | `input.apps_init` (per slot) |
|
||
| `answer = staticmethod(lambda t, apps_init: ...)` | full `input.apps_init` dict — index by app name (cross-app supported) |
|
||
| `def get_answer(self, input): ...` (override) | **you choose** — read `apps_init` for queries, `apps` for post-action |
|
||
|
||
**Migration note:** prior to this change, declarative `answer` resolved against `input.apps`. If you wrote an `AnswerTask` whose answer is supposed to reflect state *after* the Agent acts (hybrid query-after-action), switch from the declarative shorthand to an explicit `get_answer()` that reads `input.apps`.
|
||
|
||
---
|
||
|
||
## 10. Data source policy
|
||
|
||
Priority for fetching App data: **`getState()` runtime state > app offline data files > hard-coded constants**.
|
||
|
||
**Before defining a constant, check whether the data is already in `getState()`**. If it is (e.g., `recentPlays` / `likedSongs` / `contacts`), **never duplicate it as a Python module-level constant** — the copy will drift from the source.
|
||
|
||
Parameter sampling priority: **`source` > `sampler` > hard-coded `enum`**. Use `enum` only when the value domain is unrelated to environment data (e.g., a fixed enum like "last half year / last month").
|
||
|
||
If a parameter has only one meaningful value, use `default` only. Do not add a fake `source` / `sampler` just to look generalized. Numeric parameters intended to vary must declare `min`/`max`, `values`, `source`, or `sampler`.
|
||
|
||
---
|
||
|
||
## 11. Placeholder task marker
|
||
|
||
Tasks where the App functionality is incomplete must be marked with `note`:
|
||
|
||
```python
|
||
class CheckSesameCredit(AnswerTask):
|
||
note = "App page not yet implemented; needs sesame-credit page"
|
||
```
|
||
|
||
**Don't** hard-code return values in `get_answer()` to fake content (`return "59"`). Pure query answers should derive from `input.apps_init` / `input.os_init`; post-action answers should derive from `input.apps` / `input.os`.
|
||
|
||
---
|
||
|
||
## 12. Task docstring convention
|
||
|
||
A Task class docstring captures **design decisions that the code can't self-document**.
|
||
|
||
### 12.1 Write these
|
||
|
||
**1. Verdict: what counts as done, and why it can be judged correctly**
|
||
|
||
- **What is checked** — the success state in semantic terms ("WeChat new message contains the hotter city's name and its temperature")
|
||
- **Why it can be judged** (only when non-obvious) — the design point that disambiguates:
|
||
- Phrasing disambiguation: "first" means the first item in the search result list
|
||
- Parameter design: `{city}` is shared between weather and map, so consistency is intrinsic
|
||
- Sampling constraint: sampled to guarantee two books have different ratings; comparison is unambiguous
|
||
- Template anchor: the template fixes the note title, giving the judge a stable anchor
|
||
- Branch design: state which branch maps to which expected content
|
||
|
||
**2. Data injection — only when the task needs preset env state**
|
||
|
||
- "Alipay balance is randomly placed at 80%–120% of ticket price to cover both branches"
|
||
- "Target video must be un-liked; otherwise the operation has no effect"
|
||
- "Route must have ≥2 high-speed trains, otherwise 'earliest' has no meaning"
|
||
|
||
### 12.2 Don't write these
|
||
|
||
- **The implementation path of the judgment** — `check_searched()` / `redbook.first_search_note(keyword).title` etc. belong with the code itself
|
||
- **Restating the template** — anything that can be read from the template
|
||
- **Generic design intent** — what's already inferable from class name and template
|
||
|
||
### 12.3 Examples
|
||
|
||
```python
|
||
# Simple task — name + template already self-explanatory
|
||
class WeatherSummaryToWechat(BaseTask):
|
||
"""Verdict: WeChat new message contains {city}'s current weather and temperature."""
|
||
templates = ["查一下{city}现在天气怎么样，发给微信好友{contact}"]
|
||
|
||
# Disambiguation design
|
||
class WeatherFilterNonRainyDays(BaseTask):
|
||
"""The template fixes the note title '适合出行的日子' as a stable anchor for the judge.
|
||
Verdict: note title matches; body contains every non-rainy date."""
|
||
templates = ["查{city}未来五天天气，把不下雨的日期记在笔记里，标题写'适合出行的日子'"]
|
||
|
||
# Needs data injection
|
||
class RailwayPriceVsBalance(AnswerTask):
|
||
"""Verdict: the agent's 'enough'/'not enough' answer agrees with the actual comparison.
|
||
Injection: random Alipay balance in 100–1000 to cover both branches."""
|
||
templates = ["查{date}从{from}到{to}最便宜的高铁票多少钱，再看看支付宝余额够不够买"]
|
||
```
|
||
|
||
### 12.4 Rules
|
||
|
||
1. **Document only what the code can't self-document** — if the name + template + parameters explain it, don't add words
|
||
2. **State what and why, not how** — describe the success condition and the reason it's judgeable; don't restate `check_goals`
|
||
3. **Keep it short** — 1–5 lines. If the logic needs paragraphs of explanation, the task itself probably needs simplification
|
||
4. **Update the docstring when you change the task** — a stale docstring is worse than none
|
||
|
||
---
|
||
|
||
## 13. Common pitfalls
|
||
|
||
Every entry below comes from a real past bug.
|
||
|
||
### 13.1 Pipeline awareness: state may already be enriched
|
||
|
||
App state goes through `defaults.json → data/index.ts (enrichment) → store → bench_env state`. Many apps enrich records in `data/index.ts` or store actions (e.g., Alipay's `enrichTransferRecord` fills in `category` / `kind` / `displayTitle`), so those fields **already exist** in the state the judge sees.
|
||
|
||
**Typical mistake**: writing 80 lines of if-else regex in `app.py` to infer `category`, when `category` is already a field on the record.
|
||
|
||
**Rule**: before writing judge logic, check whether the target field is already present in `data/index.ts` or `state.ts`. **Read it directly; don't re-derive it in Python**.
|
||
|
||
### 13.2 `get_answer()` return type must match `match_value` and Agent phrasing habits
|
||
|
||
The Agent is a VLM — it reads the screen and replies in its own words. `match_value` has different semantics per type. If the two don't line up, judging is wrong.
|
||
|
||
| Pitfall | Wrong | Correct |
|
||
|---|---|---|
|
||
| Trailing zeros | `f"{total:.2f}"` → `"278.20"` | `f"{round(total, 2):g}"` → `"278.2"` |
|
||
| Month format | `"2026-01"` | `f"{year}年{month}月"` |
|
||
| Returns str instead of number | `return str(count)` | `return count` (int) |
|
||
| Time/duration via `match_value` | `build_answer_checks({"历时": "0小时59分"}, answer)` | Use `match_duration` / `match_time` in `check_goals` |
|
||
|
||
**Principle**: think from the Agent's perspective — how would it phrase the answer after seeing the screen? The return type of `get_answer()` must let `match_value` match reasonable variants.
|
||
|
||
### 13.3 CriteriaTask must ensure initial state ≠ target state
|
||
|
||
If `criteria`'s target value happens to equal the initial value in `defaults.json`, the task passes without the Agent doing anything.
|
||
|
||
| Scenario | Needs `_invert_criteria` |
|
||
|---|---|
|
||
| Target varies (toggle / enum parameter) | **Yes** |
|
||
| Target fixed, but equals initial value | **Yes** |
|
||
| Target fixed, initial value is already the opposite | No |
|
||
|
||
### 13.4 `expected_changes` must cover all side effects
|
||
|
||
Undeclared side effects cause `clean=False` warnings.
|
||
|
||
| Operation | Easily-missed fields |
|
||
|---|---|
|
||
| Viewing messages | `conversations.lastReadAt` |
|
||
| Transfer / payment | `transferDraft` / `transferReceipt` / `lastPaymentHint` |
|
||
| Search | `searchHistory` / `billSearchHistory` |
|
||
| Favorite / like | `favoriteIds` / `likedIds` |
|
||
|
||
**Rule**: run the task once in the UI, diff before vs. after, and add every changed path.
|
||
|
||
### 13.5 `_prepare()` must not hard-code data replacements
|
||
|
||
`_prepare()` exists to configure initial state when `defaults.json` defaults are insufficient. **It is not** a place to hard-code data to "control difficulty".
|
||
|
||
```python
|
||
# ❌ Defaults are fine; still injects a hard-coded copy
|
||
async def _prepare(self, env):
|
||
await env.set_state({"apps": {"spotify": {"likedSongs": [
|
||
{"id": "song_001", "artist": "周杰伦", ...},
|
||
...
|
||
]}}})
|
||
|
||
# ✅ Use defaults.json; if defaults are wrong, fix defaults.json
|
||
```
|
||
|
||
**Rule**:
|
||
|
||
1. Prefer the data already in `defaults.json`
|
||
2. If defaults don't fit, **stop and raise the issue** — change `defaults.json`, don't silently overlay hard-coded data in `_prepare()`
|
||
3. When injection truly is needed, push the schema-coupled construction down to a `prepare_state_with_*` helper in the corresponding `app.py`
|
||
|
||
### 13.6 Route criteria must account for query params
|
||
|
||
Routes often contain query params (e.g., `/chat?id=conv_p_10&type=person`); a hard-coded `criteria` of `"/chat"` fails.
|
||
|
||
**Rule**:
|
||
|
||
- If the task goal is not "navigate to a specific page", **do not put `route` in `criteria`** — the state change itself is enough
|
||
- If you do need a route check, confirm the actual route format and use an appropriate matcher
|
||
|
||
---
|
||
|
||
## 14. Final checklist
|
||
|
||
Use this checklist when adding or modifying a task. PR reviews compare against it.
|
||
|
||
### Design & base class
|
||
|
||
- [ ] **Merge vs split**: for multi-variant tasks, are the parameters orthogonal and under the same interaction? If params are coupled or interactions differ, split them.
|
||
- [ ] **Base class**: using the most suitable standard class? `CriteriaTask` / `AnswerTask` before `BaseTask`.
|
||
- [ ] **Declarative first**: anything expressible via `answer = ".path"` / `criteria = {"key": "value"}` is declared, not coded.
|
||
- [ ] **Class name**: accurately reflects the goal.
|
||
- [ ] **`objective` correct**: matches the actual behavior.
|
||
|
||
### Metadata
|
||
|
||
- [ ] **Four axes + capabilities**: `scope` / `objective` / `composition` / `difficulty` / `capabilities` all set.
|
||
- [ ] **Difficulty calibrated**: matches the Golden Steps range.
|
||
- [ ] **`max_steps` valid if set**: omit it for the difficulty default, or set exactly one of `15`, `30`, `45`, `60`.
|
||
- [ ] **`capabilities` only core**: 1–4 entries; nav and other prerequisite steps not labeled.
|
||
|
||
### Template
|
||
|
||
- [ ] **Intent, not steps**.
|
||
- [ ] **No "view" without output**: only `query` tasks use "view"; `operate` uses "do this for me".
|
||
- [ ] **No answer leak**: `AnswerTask` templates don't carry answer params (`{phone}` / `{income}`).
|
||
- [ ] **`{param}` positioned**: rendered sentence reads naturally for bool with `values`.
|
||
- [ ] **L3/L4 has variants**: 2+ templates with identical expected answer content.
|
||
|
||
### CriteriaTask
|
||
|
||
- [ ] **`criteria` is a class variable** (not `@property`).
|
||
- [ ] **Parameterize with `"{param}"`**.
|
||
- [ ] **Value mapping via `values` dict** (no `_XXX_MAP`).
|
||
- [ ] **Array lookup uses `[field={param}]`** (no hand-written `check_goals`).
|
||
- [ ] **Missing target checks use `None` deliberately**: `[field={param}] = None` means the resolved path is absent or `None`.
|
||
- [ ] **Parameter semantics match store** (no `not` to flip).
|
||
- [ ] **Initial state ≠ target**: toggle/enum tasks use `_post_sample` + `_invert_criteria`?
|
||
- [ ] **Custom `check_goals()` preserves criteria** via `super().check_goals(input)`, or the task uses `BaseTask`.
|
||
- [ ] **Array/filter paths don't rely on `_invert_criteria`**; write custom `_post_sample()` when the path contains `[`.
|
||
|
||
### AnswerTask
|
||
|
||
- [ ] **Prefer `answer` class var**: path / dict-of-paths / callable / literal.
|
||
- [ ] **`get_answer()` only when necessary**.
|
||
- [ ] **`get_answer()` returns ground truth, not judgment logic**.
|
||
- [ ] **Ties / synonyms use `re.Pattern`** (not a hard-coded string).
|
||
- [ ] **Answer-computation pushed into App answer methods** (one-line `get_answer()`).
|
||
- [ ] **Boolean queries**: detect negation before affirmation inside `check_goals`.
|
||
- [ ] **Date matching uses `date_match_labels(date, input.os)`** (pass `os_state` for relative dates).
|
||
- [ ] **Time/duration uses `match_time` / `match_duration`** (not `match_value` substring).
|
||
- [ ] **Don't validate non-answer content via `input.answer`** (messages/operations should check state).
|
||
|
||
### check_goals
|
||
|
||
- [ ] **Every check has `passed`**.
|
||
- [ ] **expected / actual are readable** (no `expected=True, actual=None`).
|
||
- [ ] **Only check Agent behavior** (no env preconditions or out-of-control conditions).
|
||
- [ ] **One semantic goal = one check** (don't split into field-level checks).
|
||
- [ ] **High-frequency patterns use App `check_*`** (task-specific logic stays inline).
|
||
- [ ] **Cover the template's implicit constraints** ("send to Moments" implies no image attachments).
|
||
- [ ] **`operate` tasks only check final state** (no mid-process checks; navigation tasks are an exception).
|
||
- [ ] **True dependencies fill placeholders in subsequent checks** (return list keeps a stable length).
|
||
|
||
### Judging reliability
|
||
|
||
- [ ] **Soundness**: wrong paths can't pass; no broad-keyword or weak-trace fields letting the Agent through.
|
||
- [ ] **Completeness**: reasonable rewordings or alternative completion paths can't fail; no binding to original title / full text / fixed phrasing.
|
||
- [ ] **Checkpoint reliability**: if checking an intermediate state, it must be both necessary for the task and stably observable.
|
||
|
||
### Error handling
|
||
|
||
- [ ] **App accessor raises `ValueError`** on missing data (no `""` / `None`).
|
||
- [ ] **Task has no data-validation boilerplate** (rely on App accessor errors).
|
||
- [ ] **Env-data issue → `raise`; Agent failure → `passed=False`**.
|
||
|
||
### Defensive coding
|
||
|
||
- [ ] **No `or {}`, no `.get("key", "")`, no `"无法判断"` fallback**.
|
||
- [ ] **Index `input.apps["xxx"]` directly** (no `.get()` / `or {}`).
|
||
- [ ] **No fallback returns in `get_answer()`**.
|
||
|
||
### Parameters
|
||
|
||
- [ ] **Prefer `source` over hard-coded `enum`**.
|
||
- [ ] **Upgrade to `sampler` for filtering / dedup**.
|
||
- [ ] **`source` path actually exists in defaults/runtime state**.
|
||
- [ ] **Single-value params use `default` only** (no fake `source` / `sampler`).
|
||
- [ ] **Numeric params that should vary declare a real domain** (`min`/`max`, `values`, `source`, or `sampler`).
|
||
- [ ] **Param values match the store**.
|
||
- [ ] **No module-level constants duplicating data already in `getState()`**.
|
||
|
||
### `_prepare` necessity
|
||
|
||
- [ ] **Prefer `defaults.json` defaults**.
|
||
- [ ] **If defaults are wrong, change `defaults.json`** — don't hard-code a replacement in `_prepare()`.
|
||
- [ ] **Necessary injections push down to App `prepare_state_with_*` helpers**.
|
||
|
||
### expected_changes
|
||
|
||
- [ ] **Path format correct** (single-app uses relative path; multi-app uses `appName.path`).
|
||
- [ ] **`[field=value]` filters target stable fields**; exact match wins, substring fallback is only for legacy convenience.
|
||
- [ ] **Covers every side effect** (verify by running the UI and diffing).
|
||
- [ ] **Constants defined in the corresponding `app.py`** (named `<APP>_<ACTION>_CHANGES`).
|
||
|
||
### Time API
|
||
|
||
- [ ] **Use `sim_today` / `sim_datetime`** (no `time.time()` / `date.today()` / `datetime.now()`).
|
||
|
||
### File responsibilities
|
||
|
||
- [ ] **tasks.py / defs/<Name>.py is clean**: no custom base classes, module-level helpers, App constants, or private compute methods.
|
||
- [ ] **New helpers live in `app.py`**: data methods / answer methods / `check_*` / `prepare_state_with_*` in their proper tier.
|
||
- [ ] **Cross-suite utilities live in `utils.py`**.
|
||
|
||
### Docstring
|
||
|
||
- [ ] **Task class has a docstring**.
|
||
- [ ] **States "what counts as done" and (when non-obvious) "why it's judgeable" + "what to inject"**.
|
||
- [ ] **Doesn't restate the template or describe `how`**.
|
||
|
||
### Tests
|
||
|
||
- [ ] **Each task has positive and negative cases** (see [`TASK_TESTING_GUIDE.md`](TASK_TESTING_GUIDE.md)).
|
||
- [ ] **AnswerTask positive `answer` is natural language** (not the raw ground truth).
|
||
- [ ] **Completeness check passes** (CI won't miss a new task).
|
||
|
||
### Grounded mode (only if `answer_fields` is declared)
|
||
|
||
- [ ] **Right field type**: prefer `choice` over `text`; pure numbers use `number`.
|
||
- [ ] **Right matcher**: `"time"` for time, `"date"` for date, `"duration"` for duration; otherwise auto by type.
|
||
- [ ] **Override `get_expected_response()` when `get_answer()` returns `re.Pattern`** — grounded mode needs exact values.
|
||
- [ ] **Override `get_expected_response()` when `get_answer()` returns a dict but `answer_fields` count doesn't match**.
|
||
- [ ] **repeatable field**: `get_expected_response()` returns `[[v1, v2, ...]]` (outer list = 1 entry, inner = full list).
|
||
- [ ] **hint cross-validates with check**: hint sample value must match both the task's natural expectation and what the check actually accepts.
|
||
- [ ] **Custom `check_goals` compatible with both modes**: in grounded mode, `input.answer` is the `", "`-joined AnswerSheet values.
|
||
- [ ] **Multi-field same-type false-positive risk**: such tasks shouldn't write custom `check_goals`; let the framework go via Path A.
|
||
|
||
See [`GROUNDED_MODE.md`](GROUNDED_MODE.md).
|
||
|
||
### Code-vs-docs
|
||
|
||
- [ ] **APIs referenced in examples / docs must exist** — every App method, constant, parameter referenced in an example must be findable in the matching `app.py`.
|
||
|
||
---
|
||
|
||
## Appendix: why custom base classes are banned
|
||
|
||
Historically, several suites (railway12306, bilibili, etc.) used custom base classes like `_RailwayBaseTask` / `_RailwayMixin` to deduplicate. They created three problems:
|
||
|
||
1. **Coupling propagated implicitly through the base** — changing one task could break others
|
||
2. **Tasks were no longer self-contained** — reading a single task required understanding the base
|
||
3. **Inconsistent abstraction levels** — some tasks went through the base, others didn't, drifting the convention
|
||
|
||
**Hard rule**: every task inherits one of `BaseTask` / `CriteriaTask` / `AnswerTask` directly. To reduce duplication, push shared logic down into App `check_*` methods; don't introduce intermediate base classes.
|