wehub-resource-sync
426e9eeabd
Benchmark Bridge Tests / benchmark (bunx @biomejs/biome check packages/lifeops-bench/src, benchmark-lint) (push) Waiting to run
Benchmark Bridge Tests / benchmark (bunx vitest run --config packages/lifeops-bench/vitest.config.ts --root packages/lifeops-bench --passWithNoTests, benchmark-tests) (push) Waiting to run
Build Agent Image / build-and-push (push) Waiting to run
Chat shell gestures / Chat shell gesture + parity e2e (push) Waiting to run
Cloud Gateway Discord / Test (push) Waiting to run
Cloud Gateway Webhook / Test (push) Waiting to run
Cloud Tests / lint-and-types (push) Waiting to run
Cloud Tests / unit-tests (push) Waiting to run
Cloud Tests / integration-tests (push) Waiting to run
Cloud Tests / e2e-tests (push) Blocked by required conditions
CodeQL Advanced / Analyze (javascript-typescript) (push) Waiting to run
Deploy Apps Worker (Product 2) / Determine environment (push) Waiting to run
Deploy Apps Worker (Product 2) / Deploy apps worker to apps-control host (${{ needs.determine-env.outputs.environment }}) (push) Blocked by required conditions
Deploy Eliza Provisioning Worker / Determine environment (push) Waiting to run
Deploy Eliza Provisioning Worker / Deploy worker to Hetzner host (${{ needs.determine-env.outputs.environment }} @ ${{ needs.determine-env.outputs.deployment_sha }}) (push) Blocked by required conditions
Dev Smoke / Classify changed paths (push) Waiting to run
Dev Smoke / bun run dev onboarding chat (push) Blocked by required conditions
Dev Smoke / Vite HMR dependency-level smoke (push) Blocked by required conditions
Electrobun Submodule Guard / electrobun gitlink is fetchable (push) Waiting to run
Publish @elizaos/example-code / check_npm (push) Waiting to run
Publish @elizaos/example-code / publish_npm (push) Blocked by required conditions
Publish @elizaos/plugin-elizacloud / verify_version (push) Waiting to run
Publish @elizaos/plugin-elizacloud / publish_npm (push) Blocked by required conditions
Sandbox Live Smoke / Sandbox live smoke (push) Waiting to run
Snap Build & Test / Build Snap (amd64) (push) Waiting to run
Snap Build & Test / Build Snap (arm64) (push) Waiting to run
supply-chain / sbom (push) Waiting to run
supply-chain / vulnerability-scan (push) Waiting to run
Build, Push & Deploy to Phala Cloud / build-and-push (push) Waiting to run
Test Packaging / Validate Packaging Configs (push) Waiting to run
Test Packaging / PyPI on Python ${{ matrix.python }} (push) Waiting to run
Test Packaging / Pack & Test JS Tarballs (push) Waiting to run
Test Packaging / elizaos CLI global-install smoke (node + bun) (push) Waiting to run
UI Fixture E2E / ui-fixture-e2e (push) Waiting to run
UI Fixture E2E / fixture-e2e (push) Waiting to run
UI Story Gate / story-gate (push) Waiting to run
vault-ci / test (macos-latest) (push) Waiting to run
vault-ci / test (ubuntu-latest) (push) Waiting to run
vault-ci / test (windows-latest) (push) Waiting to run
vault-ci / app-core wiring tests (push) Waiting to run
verify-patches / verify patches/CHECKSUMS.sha256 (push) Waiting to run
Voice Benchmark Smoke / voice-emotion fixture smoke (push) Waiting to run
Voice Benchmark Smoke / voiceagentbench fixture smoke (push) Waiting to run
Voice Benchmark Smoke / voicebench-quality unit smoke (push) Waiting to run
Voice Benchmark Smoke / voicebench TypeScript unit (no audio) (push) Waiting to run
Voice Benchmark Smoke / voice bench smoke summary (push) Blocked by required conditions
Windows CI / windows ([bun run --cwd packages/app-core test bun run --cwd packages/elizaos test bun run --cwd packages/cloud/shared test], app-and-cli) (push) Waiting to run
Windows CI / windows ([bun run --cwd packages/scenario-runner test bun run --cwd packages/vault test bun run --cwd packages/security test bun run --cwd plugins/plugin-coding-tools test], framework-packages) (push) Waiting to run
Windows CI / windows ([bun run --cwd plugins/plugin-elizacloud test bun run --cwd plugins/plugin-discord test bun run --cwd plugins/plugin-anthropic test bun run --cwd plugins/plugin-openai test bun run --cwd plugins/plugin-app-control test bun run --cwd plugins/pl… (push) Waiting to run
Windows CI / windows ([node packages/scripts/run-turbo.mjs run build --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/agent --concurrency=4 node packages/scripts/run-bash-linux-only.mjs scripts/verify-riscv64-buildpaths.sh node packages/scripts/run… (push) Waiting to run
Windows CI / windows ([node packages/scripts/run-turbo.mjs run typecheck --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/cloud-shared --concurrency=4 bun run --cwd packages/core test bun run --cwd packages/shared test], core-runtime, 75) (push) Waiting to run
Test Packaging / Build & Test PyPI Package (push) Waiting to run
Voice Workbench / headless workbench (mocked backends) (push) Has been cancelled
Voice Workbench / real acoustic lane (nightly, provisioned only) (push) Has been cancelled
ci / test (push) Has been cancelled
ci / lint-and-format (push) Has been cancelled
ci / build (push) Has been cancelled
ci / dev-startup (push) Has been cancelled
gitleaks / gitleaks (push) Has been cancelled
Markdown Links / Relative Markdown Links (push) Has been cancelled
Quality (Extended) / Homepage Build (PR smoke) (push) Has been cancelled
Quality (Extended) / Comment-only diff guard (push) Has been cancelled
Quality (Extended) / Format + Type Safety Ratchet (push) Has been cancelled
Quality (Extended) / Develop Gate (secret scan + UI determinism) (push) Has been cancelled
Quality (Extended) / Develop Gate (lint) (push) Has been cancelled
81 lines
3.2 KiB
Markdown
81 lines
3.2 KiB
Markdown
# `@elizaos/plugin-computeruse`
|
|
|
|
Desktop automation plugin for elizaOS agents — screenshots, mouse /
|
|
keyboard control, browser CDP automation, window management, clipboard,
|
|
and the OCR provider registry that other plugins contribute to.
|
|
|
|
Ported from
|
|
[`coasty-ai/open-computer-use`](https://github.com/coasty-ai/open-computer-use)
|
|
(Apache 2.0).
|
|
|
|
## Boundary with `@elizaos/plugin-vision`
|
|
|
|
This plugin owns the OS surfaces:
|
|
|
|
- screen / display capture (`src/platform/capture.ts`,
|
|
`src/platform/displays.ts`,
|
|
`ComputerUseService.captureScreen()`),
|
|
- input + windows + clipboard + accessibility,
|
|
- the OCR provider registries — `OcrProvider` (line-level) and
|
|
`CoordOcrProvider` (hierarchical with absolute coords), defined in
|
|
`src/mobile/ocr-provider.ts`.
|
|
|
|
`@elizaos/plugin-vision` owns the camera pipeline, scene description
|
|
via `runtime.useModel(IMAGE_DESCRIPTION)`, the screen tiler, the
|
|
detector pipeline (faces / people / objects), and the OCR
|
|
implementations themselves. plugin-vision *consumes* capture from this
|
|
plugin via `runtime.getService("computeruse")` and *contributes* the
|
|
hierarchical OCR adapter into this plugin's `registerCoordOcrProvider`
|
|
seam at boot.
|
|
|
|
Both seams are runtime feature-detected — neither package depends on
|
|
the other.
|
|
|
|
## Enabling
|
|
|
|
- Config: `features.computeruse: true`
|
|
- Env: `COMPUTER_USE_ENABLED=1`
|
|
|
|
## Platform requirements
|
|
|
|
| OS | Capture | Input |
|
|
|----|---------|-------|
|
|
| macOS | `screencapture` (built-in) | `cliclick` (`brew install cliclick`), AppleScript |
|
|
| Linux | `import` (ImageMagick) / `scrot` | `xdotool` (`sudo apt install xdotool`) |
|
|
| Windows | PowerShell + `System.Drawing` | PowerShell |
|
|
| Browser | — | `puppeteer-core` + Chrome / Edge / Brave |
|
|
|
|
## Surface
|
|
|
|
- **Actions** — `COMPUTER_USE` (canonical screenshot / click / key /
|
|
scroll / etc.), `WINDOW` (list / focus / arrange / move /...), and
|
|
`COMPUTER_USE_AGENT` (high-level goal-driven autonomous desktop loop:
|
|
Brain → Cascade → dispatch up to `maxSteps` iterations).
|
|
Subactions of `COMPUTER_USE` and `WINDOW` are promoted to virtual
|
|
top-level actions (e.g. `COMPUTER_USE_CLICK`, `WINDOW_FOCUS`) so the
|
|
planner picks a specific verb directly from the catalogue.
|
|
- **Services** — `ComputerUseService` (`serviceType = "computeruse"`)
|
|
and `VisionContextProvider`.
|
|
- **Providers** — `computerStateProvider`, `sceneProvider`.
|
|
- **Routes** — approval inbox + SSE stream + approval-mode toggle under
|
|
`/api/computer-use/...`.
|
|
|
|
## File operations + shell
|
|
|
|
File operations live on the FILE action; shell / terminal access lives
|
|
on the SHELL action. They are **not** exposed by this plugin.
|
|
|
|
## Further reading
|
|
|
|
- [`docs/MULTI_MONITOR.md`](./docs/MULTI_MONITOR.md) — multi-display
|
|
capture and coordinate translation.
|
|
- [`docs/SCENE_BUILDER.md`](./docs/SCENE_BUILDER.md) — how windows,
|
|
a11y, screen, and OCR are composed into a single `Scene`.
|
|
- [`docs/IOS_CONSTRAINTS.md`](./docs/IOS_CONSTRAINTS.md) /
|
|
[`docs/ANDROID_CONSTRAINTS.md`](./docs/ANDROID_CONSTRAINTS.md) —
|
|
honest scope on mobile.
|
|
- [`docs/MOBILE_ASSISTANT_ROUTING.md`](./docs/MOBILE_ASSISTANT_ROUTING.md)
|
|
— mobile request routing.
|
|
- [`docs/AOSP_SYSTEM_APP.md`](./docs/AOSP_SYSTEM_APP.md) — AOSP
|
|
system-app deployment notes.
|