jackwener--opencli/docs/design/mercury-fix-mvp.md

Mercury Dropdown MVP

This is the short-term implementation slice for the Mercury expense category failure. It is intentionally smaller than the full browser-agent-runtime roadmap.

Problem

Mercury-style category controls are usually custom React dropdowns. The trigger and option are not native <select> elements. Libraries such as Radix UI, Material UI, and shadcn commonly open or commit selection on pointer/mouse down or up events.

OpenCLI's generic browser click currently calls DOM el.click() first. That only dispatches a click event, so OpenCLI can return success while the dropdown never opened or the option never committed.

agent-browser succeeds in this class mainly because its click path sends real CDP mouse events. It still uses the normal loop:

snapshot
click trigger
snapshot
click option
snapshot or get value

It does not have a general one-shot custom-dropdown choose command.

MVP Scope

0. Extension CDP Capability Check

The MVP depends on Chrome debugger CDP passthrough. Unlike agent-browser, OpenCLI usually reaches Chrome through the extension, so command availability must be explicit.

Current status:

CDP command	Extension status	MVP action
Input.dispatchMouseEvent	Allowed in extension/src/background.ts CDP passthrough and already used by nativeClick.	No extension change for PR 1.
Accessibility.getFullAXTree	Allowed in CDP passthrough.	No extension change for AX fetch.
DOM.getBoxModel	Allowed in CDP passthrough.	No extension change for CDP point measurement.
DOM.describeNode	Not currently in the CDP passthrough allowlist.	Add to allowlist before AX iframe/subtree work.

Therefore PR 1 can ship without extension changes. PR 2 must either avoid DOM.describeNode or include the small extension allowlist addition in the same PR. If AX re-resolution needs DOM.resolveNode, that command must be added and tested at the same time rather than assumed.

1. CDP Mouse Primary

Change generic browser click to:

1. resolve target,
2. scroll into view and measure a non-zero target box,
3. send CDP Input.dispatchMouseEvent sequence: mouseMoved -> mousePressed -> mouseReleased,
4. fall back to DOM el.click() only when native click is unavailable or the target has no usable point.

Required output behavior:

• keep existing success shape,
• add diagnostics only as additive fields if needed,
• never report success from JS click before trying CDP when CDP is available.

This applies only to click-like actions: browser click and later browser dblclick. It does not change browser select; native <select> continues to use the existing JS option setter/change-event path.

2. Real Component Fixture

Add a local browser fixture that records event order and selected value.

Minimum cases:

• Radix/shadcn-like select:
  • trigger opens on pointerdown,
  • option selects on pointerup or mousedown,
  • menu is rendered in a portal.
• Material UI-like autocomplete:
  • combobox input opens popup,
  • option list is outside the trigger subtree.
• Native <select> remains covered by existing browser select.

Pass condition:

• Before PR 1, run and record baseline pass rate. At least one custom fixture case must fail on the DOM el.click() path.
• After PR 1, custom fixture pass rate must be at least N-1/N, where N is the number of custom dropdown cases in the fixture.
• OpenCLI verifies the selected text/value changed.

3. AX Snapshot Prototype

Add an AX-backed snapshot/ref-map prototype behind a non-breaking option or internal test helper.

Required data per ref:

type BrowserRef = {
  ref: string;
  backendNodeId?: number;
  role: string;
  name: string;
  nth?: number;
  frame?: { frameId?: string; sessionId?: string; url?: string };
};

Required behavior:

• build refs from Accessibility.getFullAXTree,
• use backendDOMNodeId as the fast path for action resolution,
• if that id is stale, re-query the AX tree by role/name/nth,
• keep current DOM snapshot text output stable until the AX path is proven.

This is the part that should learn most directly from agent-browser.

Success metric:

• stale-ref recovery fixture must pass at least 9/10 repeated React re-render runs by resolving through AX role/name/nth.

4. Native Type/Fill Normalization

Review existing nativeType and fillText paths and make them consistent with the native-input backend:

• focus through CDP when possible,
• use Input.insertText for printable text,
• keep fill exact-replacement semantics,
• keep existing verification as the authority for fill success.

Do not expand this into a full actionability rewrite in the MVP.

Non-Goals

• No full Playwright actionability pipeline.
• No broad Playwright API clone.
• No general natural-language act.
• No one-shot browser choose in this MVP.
• No route/HAR/mock/trace-viewer surface.

browser choose can be considered after this MVP if measurements show that the snapshot/click/snapshot/click loop is reliable but still too expensive for agents.

PR Breakdown

PR 0: Extension Allowlist For AX Prototype

Only needed if PR 2 uses commands that are not already allowed.

• add DOM.describeNode to CDP passthrough allowlist,
• add DOM.resolveNode only if the implementation needs it,
• add extension tests that blocked commands remain blocked and allowed AX/DOM commands pass.

PR 1: CDP-Primary Click

• flip generic click to CDP-first,
• keep JS fallback,
• add event-order fixture tests,
• run browser/unit gates and full adapter tests,
• list every new adapter failure compared with main and fix them before merge.

This corresponds to the immediate Mercury reliability fix.

PR 2: AX Ref Prototype

• add AX tree fetch through existing page.cdp,
• create internal BrowserRef map,
• implement cached backend id resolution plus role/name/nth fallback,
• add stale React re-render fixture.

Keep this additive. Do not replace browser state default in the same PR.

Frame-aware routing is not part of the MVP. It moves to Phase 1 because the Mercury exit criteria do not require iframe support.

Status after implementation:

• PR 1 shipped CDP-primary click and component fixtures.
• PR 2 shipped opt-in browser state --source ax, backend-node ref clicks, and stale role/name/nth recovery.
• Phase 1 same-origin iframe refs shipped for AX snapshots; cross-origin session routing remains deferred.
• Phase 1 metrics shipped as browser state --compare-sources so AX default promotion can be decided from measured DOM-vs-AX data.
• Phase 1 visual refs shipped as browser screenshot --annotate, giving agents a screenshot whose visible labels map back to normal DOM [N] refs.

Phase 1 Follow-Up

• carry same-origin frameId and cross-origin session id when available,

• route click/fill/type by ref frame context,

• return typed frame_unreachable when not possible,

• add iframe fixture.

• decide whether AX becomes default observation or an explicit --source ax,

• update skills/opencli-browser/SKILL.md,

• add troubleshooting docs and fixture examples,

• record manual Mercury or Mercury-equivalent validation.

Compatibility Plan

Change	Compatibility risk	Mitigation
CDP click primary	Event order changes from synthetic click to real mouse sequence.	This is desired for dropdowns. PR 1 must run full adapter tests; all new failures must be listed and fixed before merge. Keep OPENCLI_BROWSER_CLICK=js as emergency plan B for one release, not as a substitute for fixing tests.
AX refs	Ref internals change; text output should not.	Add AX map internally first; preserve visible state format.
Stale-ref recovery	A stale ref may now resolve to a new node with same role/name/nth.	Only use fallback for refs, not arbitrary CSS selectors; include diagnostic field when recovery happens.
Frame routing	Actions may reach iframe elements that previously failed.	Phase 1 only. Add typed errors for unsupported frames instead of silent fallback.
Native select	Risk of accidental behavior change if "click-like" is interpreted broadly.	Out of scope for PR 1. browser select keeps JS option setter/change-event behavior.

Exit Criteria

• A custom dropdown fixture that depends on pointer/mouse events passes through browser click.
• A portal-rendered option can be selected with the normal snapshot/click/snapshot/click loop.
• A stale React-ref fixture recovers through AX role/name/nth.
• Existing browser tests and adapter tests pass.
• Documentation tells agents the correct current recipe and does not promise choose until it exists.

Quantitative gates:

• Custom dropdown fixtures: record baseline before PR 1, then pass at least N-1/N custom cases after PR 1.
• Stale-ref recovery: pass at least 9/10 repeated re-render runs after PR 2.
• Adapter compatibility: zero unexplained new full-adapter-test failures before a default behavior PR merges.

Manual SaaS check:

• After Phase 0, @opencli-质量官 runs Mercury, Brex, and Linear manually when credentials/access are available.
• Pass means the workflow can select the relevant category/field and save or commit the form state.
• For each site, also run opencli browser state --compare-sources on the form page and record sources.dom.refs, sources.ax.refs, frame_sections, approx_tokens, elapsed_ms, and any per-source error.
• Failure does not block MVP retroactively, but each failure must be recorded as a Phase 1 backlog item with observed command sequence and failure reason.

Validation Commands

npm run build
npm run typecheck
npm test -- --run src/browser src/cli.test.ts
npm run check:typed-error-lint
npm run check:silent-column-drop

Before merging a default behavior change:

npm test -- --run

If full tests are too slow in the review loop, run full adapter tests at least once before merge and report any difference from main.