chore: import upstream snapshot with attribution

docs/design/browser-agent-runtime.md

@@ -0,0 +1,503 @@
+# Browser Agent Runtime
+
+OpenCLI browser should become reliable for unknown SaaS workflows without
+becoming a Playwright clone. The target is an agent-native runtime:
+
+- compact observation that gives an agent the right refs in one call,
+- refs that survive common React re-render/stale-node cases,
+- real browser input events for user-like interactions,
+- a small command surface that reduces calls only when the abstraction is
+  proven.
+
+This design was triggered by a customer report: Mercury expense category
+dropdowns worked with `agent-browser + Claude Code`, but OpenCLI often failed
+to select the category.
+
+## Source Findings
+
+We compared OpenCLI against `vercel-labs/agent-browser` at source level, not
+only from documentation.
+
+### What agent-browser Actually Does
+
+The reliable dropdown behavior is not from a full Playwright actionability
+pipeline or a magic custom-select command.
+
+- `cli/src/native/interaction.rs`: `click` resolves the target center and sends
+  CDP `Input.dispatchMouseEvent` as `mouseMoved`, `mousePressed`,
+  `mouseReleased`. It does not run Playwright's full visible/stable/enabled/
+  receives-events pipeline.
+- `cli/src/native/interaction.rs`: `select_option` only handles native
+  `<select>` by mutating selected options and dispatching `change`.
+- `cli/src/native/snapshot.rs`: snapshots come from
+  `Accessibility.getFullAXTree`, not a DOM walk. Refs carry accessibility
+  role/name and backend node id.
+- `cli/src/native/element.rs`: ref resolution first tries the cached
+  `backendDOMNodeId`; if stale, it re-queries the AX tree by role/name/nth.
+  This is the important stale-ref recovery mechanism.
+- `cli/src/native/snapshot.rs` and `cli/src/native/actions.rs`: iframe refs
+  carry frame context, including dedicated sessions for cross-origin iframes
+  when CDP exposes them.
+- `cli/src/native/actions.rs`: semantic locators such as `getbyrole` are
+  simple JS queries that mark an element with `data-agent-browser-located`,
+  then delegate to the same action path.
+- `skill-data/core/SKILL.md`: the intended agent loop remains
+  `snapshot -> click/fill/type by @ref -> snapshot again`.
+
+Implication: for Mercury-style custom dropdowns, the minimum necessary fix is
+real CDP input plus better refs/observation. A one-shot `choose` command is a
+later optimization, not the root cause fix.
+
+### Current OpenCLI Gaps
+
+- `src/browser/base-page.ts` and `src/browser/target-resolver.ts`: generic
+  `browser click` currently resolves an element and calls DOM `el.click()`
+  first. CDP native click is only a fallback when JS click fails. Radix/MUI/
+  shadcn controls often open/select on `pointerdown`, `mousedown`, `mouseup`,
+  or `pointerup`, so `el.click()` can report success while the UI did not
+  change.
+- `src/browser/dom-snapshot.ts`: OpenCLI observation is DOM-based. It emits
+  useful compact refs, but refs do not have AX role/name/nth fallback semantics
+  and are weaker across re-rendered portals.
+- OpenCLI already has the right low-level plumbing: `IPage.cdp`,
+  `nativeClick`, `nativeType`, `nativeKeyPress`, `setFileInput`, and extension
+  CDP passthrough including `Accessibility.getFullAXTree`.
+- Extension CDP passthrough already allows `Input.dispatchMouseEvent`,
+  `Accessibility.getFullAXTree`, and `DOM.getBoxModel`. It does not currently
+  allow `DOM.describeNode`; AX subtree/iframe work must add that allowlist entry
+  before depending on it.
+- OpenCLI exposes fewer general browser primitives than agent-browser:
+  `hover`, `focus`, `check`, `uncheck`, `dblclick`, `drag`, `upload`, and
+  `wait download` are not a consistent first-class CLI surface.
+- `browser find` is currently CSS-oriented; role/name/label/text locators are
+  not first-class for agents.
+
+## Product Position
+
+OpenCLI has two browser jobs:
+
+1. deterministic adapters for known sites,
+2. a reliable browser toolbelt for unknown pages and adapter authors.
+
+Borrow from Playwright only where it improves reliability:
+
+- real pointer/mouse/keyboard/file input events,
+- selected actionability checks when they prevent silent failures,
+- strict locator ambiguity for write actions,
+- typed failure reasons.
+
+Do not copy:
+
+- Playwright's test runner, assertion DSL, trace viewer, route/HAR surface, or
+  one-to-one API list.
+
+Borrow from agent-browser:
+
+- AX-tree observation,
+- compact refs as the default agent selector,
+- role/name stale-ref re-resolution,
+- iframe-aware refs,
+- annotated screenshots mapped to refs,
+- a simple snapshot/action loop.
+
+Do not copy blindly:
+
+- broad command sprawl,
+- natural-language `act` as the primary runtime,
+- route/mock/HAR as near-term browser CLI surface,
+- claims that a custom dropdown should be solved by magic if the source system
+  itself still uses snapshot/click/snapshot/click.
+
+## Design Principles
+
+- Prefer one reliable primitive over three brittle workarounds.
+- Reduce agent calls by improving observation first. A better snapshot can
+  avoid many follow-up help/find calls without inventing high-level commands
+  prematurely.
+- Keep actions transparent. A failed action should return a branchable reason,
+  not a generic "click failed".
+- Add command surface only when it maps to a distinct browser task and reduces
+  real agent calls.
+- Keep adapter compatibility explicit. Any change to default click/type/fill
+  behavior must run full adapter tests. Escape hatches are emergency plan B, not
+  a reason to merge known regressions.
+
+## Architecture
+
+### Native Input Backend
+
+Normalize native input through a small internal backend over existing `IPage`
+capabilities:
+
+```ts
+interface NativeInputBackend {
+  click(point: Point, opts?: ClickOptions): Promise<void>;
+  dblclick(point: Point): Promise<void>;
+  hover(point: Point): Promise<void>;
+  key(key: string, modifiers?: string[]): Promise<void>;
+  insertText(text: string): Promise<void>;
+  setFiles(selector: string, files: string[]): Promise<void>;
+}
+```
+
+The Browser Bridge and direct CDP implementation can both use:
+
+- `Input.dispatchMouseEvent`,
+- `Input.dispatchKeyEvent`,
+- `Input.insertText`,
+- `DOM.setFileInputFiles`.
+
+This is the P0 fix for Mercury dropdowns: CDP mouse primary for `click`, with
+DOM `el.click()` only as fallback.
+
+This does not change `browser select`. Native `<select>` remains a separate
+operation that sets selected options and dispatches `change`, matching both
+OpenCLI's current behavior and agent-browser's `select_option` behavior.
+
+### AX Observation And Refs
+
+Add an AX-backed observation path alongside the current DOM snapshot.
+
+Ref identity should include enough data for re-resolution:
+
+```ts
+type BrowserRef = {
+  ref: string;
+  backendNodeId?: number;
+  role: string;
+  name: string;
+  nth?: number;
+  frame?: {
+    frameId?: string;
+    sessionId?: string;
+    url?: string;
+  };
+  bbox?: { x: number; y: number; width: number; height: number };
+};
+```
+
+Resolution order:
+
+1. cached `backendNodeId`,
+2. AX re-query by role/name/nth in the same frame,
+3. existing CSS/DOM resolver fallback when the target is not a ref.
+
+The first rollout should be additive:
+
+- keep current `browser state` output stable,
+- Phase 0: add an internal or opt-in AX prototype; do not change the default
+  `browser state` text output,
+- Phase 1 decision point: either promote AX to default observation or keep it
+  opt-in,
+- decision criteria: fixture pass rate, stale-ref recovery rate, manual SaaS
+  results, adapter compatibility, and snapshot token size versus the current DOM
+  snapshot.
+
+### Frame Routing
+
+Refs from iframes need frame context so commands can act without manual frame
+switching in common cases.
+
+The route should mirror agent-browser's model:
+
+- same-origin iframe: parent session plus `frameId` params where CDP supports
+  them,
+- cross-origin iframe: dedicated attached target session when available,
+- if unsupported: typed `frame_unreachable` with the iframe ref/name/url.
+
+Do not add a global "switch frame" burden to the normal agent path unless the
+target cannot be routed automatically.
+
+### Actionability Helper
+
+Do not start with full Playwright actionability. Implement a small helper that
+prevents known silent failures:
+
+1. resolve target/ref,
+2. scroll into view when CDP supports it,
+3. measure a non-zero bounding box,
+4. dispatch native input.
+
+Boundary by phase:
+
+- MVP: scroll into view plus non-zero bounding box only.
+- Phase 1: add visible/enabled/not-editable checks if the target action needs
+  them and compatibility remains clean.
+- Phase 2: add stability and receives-events checks only if fixtures or manual
+  SaaS cases show real failures.
+
+Each added wait/check has compatibility and latency cost.
+
+Typed statuses:
+
+```ts
+type ActionStatus =
+  | 'ready'
+  | 'not_found'
+  | 'stale_ref_recovered'
+  | 'stale_ref_unresolved'
+  | 'zero_rect'
+  | 'not_visible'
+  | 'disabled'
+  | 'not_editable'
+  | 'frame_unreachable'
+  | 'native_backend_unavailable';
+```
+
+### Semantic Locators
+
+Add semantic locator support after AX refs are in place:
+
+```bash
+opencli browser click --role button --name "Submit"
+opencli browser fill --label "Email" "me@example.com"
+opencli browser get text --testid invoice-total
+```
+
+For write operations, ambiguous locators must fail with candidates. They should
+not silently choose the first match.
+
+### Command Surface
+
+Keep the surface smaller than agent-browser and Playwright.
+
+Near-term primitives:
+
+```bash
+opencli browser click <target>
+opencli browser dblclick <target>
+opencli browser hover <target>
+opencli browser focus <target>
+opencli browser check <target>
+opencli browser uncheck <target>
+opencli browser upload <target> <file...>
+opencli browser drag <source> <target>
+opencli browser wait download [pattern]
+```
+
+Keep `browser select` native `<select>` only. It should clearly return
+`not_a_select` when used on custom controls.
+
+Defer `browser choose <target> <option>` until after the MVP proves the
+snapshot/ref/action loop on Mercury-like fixtures. If added later, it should be
+a thin deterministic helper for common `combobox/listbox/menu` patterns, not a
+general AI `act` command.
+
+## Backward Compatibility
+
+| Area | Risk | Policy |
+|---|---|---|
+| `browser click` | Native CDP input triggers pointer/mouse handlers that DOM `el.click()` skipped. This is intended, but can expose sites that depended on synthetic click. | Make CDP primary with JS fallback. Run full adapter tests; all new failures are blockers unless proven unrelated. Keep an internal env escape hatch for one release only as emergency rollback support. |
+| `browser type` | CDP input can differ from DOM mutation for rich editors. | Keep existing output shape and verification. Prefer native type where already available. |
+| `browser fill` | Fill must remain exact replacement, not append typing. | Preserve exact-set semantics; native input only after clearing/focus preparation. |
+| `browser select` | Native select behavior is already established. | Do not overload for custom dropdowns. CDP-primary click does not affect `browser select`; it keeps the JS option setter/change-event path. |
+| Ref format | Agents and docs depend on compact refs. | Add metadata internally first; avoid breaking text output. |
+| Extension support | Older extensions may lack a CDP command. | Detect unsupported backend and return typed diagnostic or fallback. |
+| Adapter code | 770+ commands may rely on current page helpers. | Run targeted browser/unit tests plus full adapter tests before changing defaults. |
+
+## Milestones
+
+### Phase 0: Mercury Reliability MVP
+
+Documented in `docs/design/mercury-fix-mvp.md`.
+
+Scope:
+
+- CDP mouse primary for `browser click`,
+- Radix/shadcn/MUI-style fixture coverage,
+- AX snapshot/ref-map prototype with role/name/nth re-resolution,
+- native type/fill normalization where existing behavior is already close.
+
+Out of scope for Phase 0:
+
+- iframe/frame-aware action routing,
+- full actionability,
+- `browser choose`.
+
+Exit:
+
+- Mercury-like custom select can be completed with
+  `snapshot -> click trigger -> snapshot -> click option -> verify`.
+- No silent "clicked true but no UI event chain" failure remains for fixture
+  dropdowns.
+- Existing adapter tests do not regress.
+- Quantitative gates from `docs/design/mercury-fix-mvp.md` pass.
+
+### Phase 1: Ref And Observation Upgrade
+
+Scope:
+
+- decide AX default versus opt-in based on Phase 0 metrics,
+- AX-backed `browser state` or `browser snapshot` option,
+- ref cache that stores backend node id, role/name/nth, frame context,
+- iframe-aware action routing,
+- annotated screenshot with the same ref ids and sidecar metadata.
+
+Status after implementation:
+
+- same-origin iframe AX refs route through `Accessibility.getFullAXTree`
+  `frameId` params,
+- cross-origin iframe AX routing is best-effort. Real Chrome extension smoke
+  verified that `chrome.debugger` may not expose attachable OOPIF iframe
+  targets to extensions even after `Target.setDiscoverTargets`,
+  `Target.getTargets`, and `Target.setAutoAttach`,
+- unsupported frames degrade by omission or typed action failure rather than
+  requiring global manual frame switching.
+
+Implementation note: Phase 1 also includes a DOM-ref visual slice via
+`browser screenshot --annotate`. It refreshes current DOM refs and overlays
+visible `[N]` labels on the captured image. AX-side visual labels and richer
+sidecar metadata remain follow-up work.
+
+Exit:
+
+- Agent can act on iframe refs without manual frame selection in common cases.
+- Stale refs from simple React re-renders recover by role/name/nth.
+- Snapshot output remains compact enough for normal agent context.
+
+### Phase 2: Browser Toolbelt
+
+Scope:
+
+- `hover`, `focus`, `dblclick`, `check`, `uncheck`, `upload`, `drag`, and
+  `wait download` (implemented),
+- semantic locator options for role/name, label, text, testid,
+- structured ambiguity errors for write locators.
+
+Implementation note: semantic locators cover `browser find`, `browser click`,
+`browser get text|value|attributes`, input actions (`type`, `fill`, `select`),
+and Phase 2 action primitives (`hover`, `focus`, `dblclick`, `check`,
+`uncheck`, `upload`). `drag` uses prefixed endpoint locators (`--from-role` /
+`--to-role`, etc.) so source and target cannot be confused. Placeholder is
+folded into accessible-name matching for `--name`; a dedicated `--placeholder`
+flag can be added only if real usage shows the distinction matters.
+
+Exit:
+
+- A representative form with text, checkbox/radio, file upload, custom select,
+  and submit can be completed without `eval`.
+- Agents can usually avoid inventing CSS selectors.
+
+### Phase 3: High-Level Deterministic Helpers
+
+Only after Phase 0/1 metrics justify it:
+
+- `browser choose` for common custom select/combobox/listbox/menu controls,
+- optional `browser form inspect` and `browser form fill <json>` for structured
+  forms,
+- failure artifact bundle: screenshot, refs, target diagnostics, console/
+  network summary, and suggested next command.
+
+Do not add a free-form natural-language `act` as the primary interface.
+
+## Success Metrics
+
+Track both reliability and call count.
+
+Mercury-like custom select:
+
+- current OpenCLI baseline: often fails because click is DOM `el.click()`;
+- Phase 0 target: reliable 4-step snapshot/action loop, with fixture pass rate
+  at least `N-1/N` after recording baseline;
+- Phase 3 target, only if warranted: deterministic 1-step `choose` after
+  target discovery.
+
+Stale-ref recovery:
+
+- Phase 0 AX prototype target: at least 9/10 repeated React re-render fixture
+  runs recover through role/name/nth.
+
+Compatibility:
+
+- Any PR changing default click/type/fill behavior must run full adapter tests.
+  New failures are merge blockers unless proven unrelated to the PR.
+
+Fixture matrix:
+
+- Radix Select,
+- shadcn Select,
+- Material UI Select/Autocomplete,
+- native `<select>`,
+- checkbox/radio label retarget,
+- file upload,
+- same-origin iframe form,
+- cross-origin iframe where CDP target attach is available.
+
+Manual SaaS matrix:
+
+| Site / app | Scenario | Controls |
+|---|---|---|
+| Mercury | expense category | custom select, portal |
+| Brex | expense category / memo | custom select, text input |
+| Ramp | reimbursement category | custom select |
+| Stripe Dashboard | filters | combobox, menu |
+| Linear | issue fields | combobox |
+| Notion | property select | custom select, portal |
+| Airtable | field select | custom select, grid |
+| Workday | form dropdown | custom select, iframe risk |
+| Concur | expense form | custom select, upload |
+| GitHub | labels | combobox, portal |
+
+Manual matrix is not a CI gate at first. It is the calibration set for deciding
+whether OpenCLI is approaching agent-browser reliability.
+
+Execution process:
+
+- Phase 0 completion: @opencli-质量官 runs Mercury, Brex, and Linear when access
+  is available.
+- On each form page, collect `opencli browser state --compare-sources` so the AX
+  default decision has DOM-vs-AX metrics: refs, frame sections, approximate
+  tokens, elapsed time, and per-source errors.
+- Pass means the relevant category/field can be selected and the form state can
+  be saved or committed.
+- Failures do not block the already-shipped MVP unless they expose a regression,
+  but they become Phase 1 backlog with exact command sequence and failure
+  reason.
+
+## Test Strategy
+
+Phase 0:
+
+- unit tests for CDP-primary click fallback behavior,
+- local fixture that records pointer/mouse/click event order,
+- component fixtures for Radix/shadcn/MUI-style controls,
+- stale-ref fixture that re-renders an option after snapshot and verifies
+  AX re-resolution,
+- direct CDP and Browser Bridge paths where practical.
+
+Regression gates:
+
+```bash
+npm run build
+npm run typecheck
+npm test -- --run src/browser src/cli.test.ts
+npm run check:typed-error-lint
+npm run check:silent-column-drop
+```
+
+Before changing default action behavior, also run the adapter suite on main and
+on the branch and compare failures.
+
+## Documentation
+
+Update after Phase 0:
+
+- `skills/opencli-browser/SKILL.md`: recommend snapshot/click/snapshot/click for
+  custom dropdowns until `choose` exists.
+- Browser command help: explain that `select` is native `<select>` only.
+- Troubleshooting: explain `zero_rect`, `not_visible`, `disabled`,
+  `frame_unreachable`, `stale_ref_unresolved`, and `native_backend_unavailable`.
+- Comparison guide: OpenCLI's goal is not "Playwright in CLI form"; it is an
+  adapter-first CLI with reliable agent browser primitives.
+
+## Open Questions
+
+- Should the CDP click escape hatch be public (`--mode js`) or env-only?
+  Prefer env-only unless a real adapter regression needs user control.
+- Should AX snapshot replace DOM snapshot by default or exist as
+  `--source ax` first? Prefer opt-in/prototype first, then promote after
+  fixture and manual SaaS validation.
+- How much actionability is enough? Start with scroll/rect/visible/enabled.
+  Add stability/receives-events only with measured failures.
+- Does `browser choose` materially reduce successful Mercury workflows after
+  AX refs land? If not, do not add it.