wehub-resource-sync
278 lines
10 KiB
Markdown
278 lines
10 KiB
Markdown
# Browser Runtime API
|
||
|
||
When the simulator is running, it exposes a set of globals on `window` for debugging, scripting, and benchmark orchestration. **The agent never sees these** — they're for the human operator, task author, or benchmark runner.
|
||
|
||
The available namespaces:
|
||
|
||
| Namespace | Purpose |
|
||
|---|---|
|
||
| `window.__SIM__` | State snapshot, reset, simulator-level controls |
|
||
| `window.__OS__` | OS-level controls (open app, back, services) |
|
||
| `window.__SIM_INPUT__` | Synthesized human-style input (tap, swipe, type, …) |
|
||
| `window.__SIM_QUERY__` | Read-only DOM queries (find an element's rect by id / selector / trigger) |
|
||
| `window.__SIM_TIME__` | Simulated wall-clock control |
|
||
| `window.__SIM_LOCATION__` | Simulated geolocation control |
|
||
| `window.__SIM_FS__` | Simulated file-system access |
|
||
| `window.__SIM_MEDIA__` | Simulated media-library access |
|
||
| `window.__SIM_AI__` | Optional simulated AI service hooks |
|
||
| `window.__STORAGE_ISOLATION__` | Current storage-isolation mode metadata |
|
||
| `window.__getScrollMeta__` | Read-only scroll-container state |
|
||
|
||
You can open the browser DevTools and call any of these directly. The benchmark calls them via Playwright's `evaluate()`.
|
||
|
||
## `__SIM__` — simulator state
|
||
|
||
### `__SIM__.getState()`
|
||
|
||
Returns the entire structured state of the simulator as a JSON-serializable object.
|
||
|
||
```js
|
||
const state = window.__SIM__.getState();
|
||
// → { os: {...}, apps: { wechat: {...}, alipay: {...}, ... } }
|
||
```
|
||
|
||
This is the **only** state the benchmark trusts for judging. The full `getState()` schema, including per-app snapshots, is documented in [app-state-schema.md](app-state-schema.md). Key trees:
|
||
|
||
- `state.os.settings` — `global` / `system` / `secure` / app-specific
|
||
- `state.os.hardware` — battery, wifi, cellular, sensors
|
||
- `state.os.permissions` / `preferences`
|
||
- `state.os.providers` — contacts, sms, media (shared content)
|
||
- `state.os.clipboard` / `notifications` / `shade` — runtime singleton snapshots
|
||
- `state.os.services` — registered service snapshots other than clipboard / notifications
|
||
- `state.apps.<appId>` — each app's runtime overlay
|
||
|
||
### `__SIM__.reset()`
|
||
|
||
Clears all simulator state (localStorage + in-memory stores) and reloads the page.
|
||
|
||
```js
|
||
await window.__SIM__.reset();
|
||
```
|
||
|
||
There is also `__SIM__.resetState()` which clears state without reloading — useful when the harness has already snapshotted the tab.
|
||
|
||
### `__SIM__.setState(patch, options?)`
|
||
|
||
Merges a partial state into the live simulator. Use this to inject task-initial conditions.
|
||
|
||
```js
|
||
window.__SIM__.setState({ apps: { wechat: { user: { id: 'u_42' } } } });
|
||
window.__SIM__.setState(snapshot, { deep: true, reload: true });
|
||
```
|
||
|
||
`setState()` is synchronous. If `reload: true` is passed, it triggers `window.location.reload()` after applying the patch.
|
||
|
||
### `__SIM__.waitForData(appIds?)` / preload helpers / `warmUpAllApps()`
|
||
|
||
`waitForData(appIds?)` imports and runs app data-loader modules, with a retry for transient failures. Use it when task setup depends on app data being available before snapshotting or route setup.
|
||
|
||
`preloadAllAppStores()` and `preloadAppStores(appIds)` are currently no-ops because app stores are eagerly loaded. They remain in the runtime API for compatibility with older benchmark scripts and possible future lazy-store builds.
|
||
|
||
`warmUpAllApps()` additionally mounts each app once so its first render cost is paid up front; this is slower and usually unnecessary.
|
||
|
||
## `__OS__` — OS controls
|
||
|
||
### App lifecycle
|
||
|
||
```js
|
||
window.__OS__.openApp('wechat'); // open with current route
|
||
window.__OS__.openApp('wechat', '/chat?tab=hot'); // open at a specific deep link
|
||
window.__OS__.closeApp('wechat'); // kill the task
|
||
window.__OS__.launchApp('wechat'); // launcher-style start
|
||
window.__OS__.goHome();
|
||
window.__OS__.showRecents();
|
||
```
|
||
|
||
### Back key
|
||
|
||
```js
|
||
window.__OS__.handleBack();
|
||
// Routes through BackDispatcher according to registered priorities:
|
||
// PermissionDialog (1000) > Shade (800) > Keyboard (700) > App (100) > Launcher
|
||
```
|
||
|
||
### Reading current location
|
||
|
||
```js
|
||
window.__OS__.getAppRoute();
|
||
// → { app: 'wechat', path: '/chat?tab=hot' }
|
||
```
|
||
|
||
### Service handles
|
||
|
||
System services are accessible as sub-properties of `__OS__`. Representative calls:
|
||
|
||
```js
|
||
// Notifications
|
||
window.__OS__.notifications.push({ title: 'New message', body: 'Hi' });
|
||
window.__OS__.notifications.dismiss(id);
|
||
window.__OS__.notifications.getState();
|
||
|
||
// Keyboard
|
||
window.__OS__.keyboard.show();
|
||
window.__OS__.keyboard.hide();
|
||
window.__OS__.keyboard.isVisible();
|
||
|
||
// Permissions
|
||
window.__OS__.permissions.checkPermission('wechat', 'LOCATION');
|
||
await window.__OS__.permissions.requestPermissions('wechat', ['LOCATION', 'CAMERA']);
|
||
window.__OS__.permissions.grantPermission('wechat', 'LOCATION');
|
||
|
||
// Device preferences (battery, WiFi, brightness, …)
|
||
window.__OS__.device.getPreference('brightness');
|
||
window.__OS__.device.setPreference('brightness', 80);
|
||
window.__OS__.device.connectWifi('Home-5G');
|
||
|
||
// Quick settings
|
||
window.__OS__.quickSettings.set({ airplane: true });
|
||
```
|
||
|
||
Each service's full API surface is declared in `os/types/globals.d.ts`. The app-facing service rules are documented in [../platform/os/services/README.md](../platform/os/services/README.md).
|
||
|
||
## `__SIM_INPUT__` — synthesized input
|
||
|
||
These are the same gestures the benchmark dispatches via Playwright when an agent emits actions. Coordinates here are **CSS pixels** (viewport coordinates), not the agent's normalized `[0, 1000]`.
|
||
|
||
| Method | Signature | Notes |
|
||
|---|---|---|
|
||
| `tap(x, y, opts?)` | `(number, number, {coords?: 'css' \| 'physical'})` | Single tap |
|
||
| `doubleTap(x, y, opts?)` | `(number, number, opts?)` | Two quick taps |
|
||
| `longPress(x, y, ms?, opts?)` | `(number, number, number = 800, opts?)` | Hold (default 800 ms) |
|
||
| `swipe(start, end, opts?)` | `({x,y}|[x,y], {x,y}|[x,y], opts)` | See below |
|
||
| `drag(start, end, opts?)` | `({x,y}|[x,y], {x,y}|[x,y], opts)` | Like swipe, but with a hold and no inertia |
|
||
| `type(text, opts?)` | `(string, {clear?: boolean, perCharMs?: number})` | Types into the focused field |
|
||
| `back()` | | Equivalent to `__OS__.handleBack()` |
|
||
| `home()` | | Equivalent to `__OS__.goHome()` |
|
||
| `recent()` | | Open the recents / multitask UI |
|
||
| `enter()` | | Synthesize an Enter / return key |
|
||
|
||
Swipe options:
|
||
|
||
```js
|
||
await __SIM_INPUT__.swipe(
|
||
{ x: 200, y: 500 }, { x: 200, y: 200 },
|
||
{
|
||
ms: 300, // duration of the swipe gesture
|
||
steps: 10, // sample points
|
||
inertia: true, // continue with inertia after release
|
||
inertiaMs: 450,
|
||
inertiaDecay: 0.86,
|
||
},
|
||
);
|
||
```
|
||
|
||
### Examples
|
||
|
||
```js
|
||
__SIM_INPUT__.tap(200, 400);
|
||
await __SIM_INPUT__.type('Hello', { clear: true });
|
||
await __SIM_INPUT__.swipe([200, 500], [200, 200]);
|
||
__SIM_INPUT__.back();
|
||
```
|
||
|
||
### Coordinate space
|
||
|
||
By default, `__SIM_INPUT__` expects **CSS pixels** matching the visible viewport. If you have *physical pixels* (e.g. coordinates from a 1080×2400 image), pass `{ coords: 'physical' }`:
|
||
|
||
```js
|
||
__SIM_INPUT__.tap(540, 1200, { coords: 'physical' });
|
||
```
|
||
|
||
The mapping is computed from the active device profile and the current CSS-zoom viewport. See [`os/simInput.ts`](../../os/simInput.ts) for the exact resolution logic.
|
||
|
||
## `__SIM_QUERY__` — DOM queries
|
||
|
||
Returns positions and bounding boxes for elements **without performing any action**. Useful for task authoring, smoke tests, or chaining `query → input`.
|
||
|
||
```js
|
||
__SIM_QUERY__.getRectById('submit-btn');
|
||
__SIM_QUERY__.getRectBySelector('[data-trigger="wechat.settings.open"]');
|
||
__SIM_QUERY__.getRectByTrigger('wechat.tab.switch', { tab: 'me' });
|
||
```
|
||
|
||
Each returns either `null` (not found / not visible) or:
|
||
|
||
```js
|
||
{
|
||
rect: { x, y, width, height }, // CSS-pixel bounds
|
||
center: { x, y }, // CSS-pixel center
|
||
centerPhysical: { x, y }, // physical-pixel center (accounts for DPR + zoom)
|
||
}
|
||
```
|
||
|
||
### Composition
|
||
|
||
```js
|
||
const r = __SIM_QUERY__.getRectByTrigger('wechat.tab.switch', { tab: 'me' });
|
||
if (r) __SIM_INPUT__.tap(r.center.x, r.center.y);
|
||
```
|
||
|
||
## `__SIM_TIME__` — simulated time
|
||
|
||
The simulator's `TimeService` underlies every `now()` call in app code. By default it follows the real wall clock; the benchmark overrides it for reproducibility.
|
||
|
||
```js
|
||
__SIM_TIME__.now();
|
||
__SIM_TIME__.setSimulatedTime('2026-05-18 09:00');
|
||
__SIM_TIME__.setSimulatedTime(1747560000000);
|
||
__SIM_TIME__.setRealTime(); // revert
|
||
__SIM_TIME__.getConfig();
|
||
```
|
||
|
||
> 🕐 **App code must use `TimeService`** for displayed times, data timestamps, and judge-relevant fields. `TimeService.realNow()` is the escape hatch for measuring real elapsed time (animations, debouncing, etc.).
|
||
|
||
## `__SIM_LOCATION__` — simulated location
|
||
|
||
Replaces `navigator.geolocation` and enforces consistent GPS coordinates across runs.
|
||
|
||
```js
|
||
__SIM_LOCATION__.getCoords();
|
||
__SIM_LOCATION__.setSimulatedLocation('shanghai');
|
||
__SIM_LOCATION__.setSimulatedLocation({ latitude: 31.23, longitude: 121.47 });
|
||
__SIM_LOCATION__.simulateError(1); // 1 = permission denied, 2 = unavailable, 3 = timeout
|
||
__SIM_LOCATION__.clearError();
|
||
__SIM_LOCATION__.setRealLocation();
|
||
__SIM_LOCATION__.presets; // → { beijing: { latitude, longitude }, shanghai: {…}, tokyo: {…}, … }
|
||
__SIM_LOCATION__.getConfig();
|
||
```
|
||
|
||
> 🌐 **App code must use `LocationService`** rather than `navigator.geolocation` directly. The lint rules will reject the latter.
|
||
|
||
## Scroll observation
|
||
|
||
```js
|
||
window.__getScrollMeta__();
|
||
// → { main: { position: 120, max: 980, viewport: 600, total: 1580 }, … }
|
||
```
|
||
|
||
Auto-discovers every visible element with `data-scroll-container="<name>"` and returns its scroll state. Useful when an agent's task requires reaching a specific scroll position.
|
||
|
||
## Cheatsheet
|
||
|
||
```js
|
||
// State surgery
|
||
__SIM__.getState();
|
||
await __SIM__.reset();
|
||
|
||
// OS control
|
||
__OS__.openApp('wechat', '/chat');
|
||
__OS__.handleBack();
|
||
|
||
// Find + tap
|
||
const r = __SIM_QUERY__.getRectBySelector('[data-trigger="settings.open"]');
|
||
__SIM_INPUT__.tap(r.center.x, r.center.y);
|
||
|
||
// Type
|
||
await __SIM_INPUT__.type('Hello MobileGym 👋', { clear: true });
|
||
|
||
// Reproducibility knobs
|
||
__SIM_TIME__.setSimulatedTime('2026-05-18 09:00');
|
||
__SIM_LOCATION__.setSimulatedLocation('shanghai');
|
||
```
|
||
|
||
## Where to go next
|
||
|
||
- 📊 The full `__SIM__.getState()` schema → [app-state-schema.md](app-state-schema.md)
|
||
- 🧪 Use these APIs from inside a task setup → [../guides/add-a-task.md](../guides/add-a-task.md)
|
||
- 🤖 What the agent sees instead → [architecture.md](../platform/architecture.md#benchmark-layer-bench_env)
|