elizaos--eliza/plugins/plugin-native-activity-tracker/AGENTS.md

@elizaos/native-activity-tracker

macOS-only Swift helper that streams window/app focus and HID idle events to a typed TypeScript driver.

Purpose / role

This package is a native helper library, not an elizaOS Plugin object. It provides a TypeScript API for spawning the compiled Swift activity-collector binary and receiving a real-time stream of macOS application-focus transitions and periodic HID idle samples. Consumers (services, plugins) call startActivityCollector() directly to get the event stream. It is Darwin-only (elizaos.platforms: ["darwin"]); callers must check isSupportedPlatform() before starting the collector.

Plugin surface

This package exports a library API, not a registered elizaOS plugin. There are no actions, providers, evaluators, routes, or services registered in a plugin manifest. The public surface is:

Export	Description
isSupportedPlatform()	Returns true on Darwin; callers must gate on this before calling startActivityCollector.
startActivityCollector(options)	Spawns the Swift binary, line-parses stdout, calls onEvent per focus transition and onIdleSample per HID idle reading. Returns an ActivityCollectorHandle with .stop() and .pid.
ActivityCollectorEvent	Focus event: `{ ts, event: "activate"
ActivityCollectorIdleSample	HID idle sample: { ts, event: "hid_idle", idleSeconds } — emitted every 30 s by the Swift timer.
ActivityCollectorHandle	Handle returned by startActivityCollector: stop(): Promise<void>, `pid: number
ActivityCollectorOptions	Options bag for startActivityCollector: onEvent (required), onIdleSample, onExit, onFatal, binaryPath.
ActivityCollectorExit	Exit descriptor: { code, signal, clean, reason }.
ActivityEventKind	`"activate"
__internal	{ parseEventLine, parseCollectorLine, describeCollectorExit } — exposed for unit tests only, not stable API.

Layout

plugins/plugin-native-activity-tracker/
  src/
    index.ts                 Entire TypeScript driver: types, parseCollectorLine, startActivityCollector
    index.test.ts            Vitest unit tests for the TypeScript driver
  native/
    macos/
      activity-collector.swift   Swift source — NSWorkspace notifications + HID idle timer
      activity-collector         Compiled binary (Darwin arm64/x86_64; must be built via build:swift)
  dist/                      Compiled TypeScript output (built via build script)
  tsconfig.json
  package.json

Commands

Scripts are defined in package.json; run them from the repo root with bun run --cwd:

bun run --cwd plugins/plugin-native-activity-tracker build         # build package artifacts
bun run --cwd plugins/plugin-native-activity-tracker build:swift   # build Swift helper (Darwin only)
bun run --cwd plugins/plugin-native-activity-tracker typecheck     # TypeScript typecheck
bun run --cwd plugins/plugin-native-activity-tracker lint          # mutating Biome check
bun run --cwd plugins/plugin-native-activity-tracker lint:check    # read-only Biome check
bun run --cwd plugins/plugin-native-activity-tracker format        # write formatting
bun run --cwd plugins/plugin-native-activity-tracker format:check  # read-only formatting check
bun run --cwd plugins/plugin-native-activity-tracker test          # run package tests

Config / env vars

No environment variables are read by this package. The binaryPath option to startActivityCollector defaults to ../native/macos/activity-collector relative to dist/index.js. Override it if the binary is at a non-standard location.

macOS Accessibility permission is required at runtime for windowTitle to be populated (AX API). Without it, windowTitle is omitted from events; focus events still fire.

How to extend

Add a new event kind from the Swift side:

1. Add the new event emission to native/macos/activity-collector.swift — emit a JSON object with a new event string value.
2. Add a corresponding TypeScript interface in src/index.ts.
3. Extend ParsedCollectorLine union and parseCollectorLine() to handle the new event value.
4. Add the new callback to ActivityCollectorOptions and call it in the rl.on("line", ...) handler in startActivityCollector.
5. Recompile both: bun run build:swift then bun run build.

Add a new TypeScript-only option:

Add the field to ActivityCollectorOptions, use it inside startActivityCollector, and rerun bun run build.

Conventions / gotchas

• Darwin only. startActivityCollector throws immediately on non-Darwin. Always call isSupportedPlatform() before calling it.
• Binary must be pre-compiled. The compiled activity-collector binary is included in files[] but must be built with build:swift before first use in a fresh checkout. The build:swift step requires macOS with Xcode command-line tools (swiftc).
• stdout line protocol. The Swift binary writes one complete JSON object per line to stdout, unbuffered. Any line that fails to parse is silently dropped by the TypeScript driver (logged at debug). Stderr from the binary is logged at warn.
• Fatal threshold. The driver calls onFatal on any non-zero exit or spawn error. There is no auto-restart — the caller is responsible for restart logic.
• HID idle cadence. The Swift timer fires after 5 s initial delay then every 30 s. onIdleSample is optional and safe to ignore.
• System sleep / lock. The Swift collector emits a synthetic deactivate on willSleep, screensDidSleep, sessionDidResignActive, and com.apple.screenIsLocked, and a synthetic activate on wake/unlock. This prevents a stale frontmost app from appearing active across sleep boundaries.
• Accessibility permission. windowTitle requires macOS Accessibility permission granted to the host process. The collector proceeds without it; windowTitle is simply absent.
• No plugin manifest. This package does not export an elizaOS Plugin object and is not auto-loaded by the elizaOS plugin registry. It is a library dependency that other plugins or services import directly.
• See the root AGENTS.md for repo-wide architecture, logging, and naming rules.

⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests

The binding, repo-wide standard is AGENTS.md. Read it. Nothing in this package is done until it is proven done — a reviewer must confirm it works without reading the code, from the artifacts you attach. This applies to every feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.

• Record AND read model trajectories. Capture the actual inputs and outputs of the model from a live LLM — not the deterministic proxy, not a mock: the prompt, the providers/context, the raw model output, every tool/action call, and the result. Then open the trajectory and review it by hand. A captured-but-unread trajectory is not evidence (packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>).
• Real, full-featured E2E — no larp. Every feature ships detailed end-to-end tests that drive the real path end to end. Not the happy "front door" only: cover error paths, edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that asserts against a mock/stub/fixture standing in for the thing under test does not count. If the real model/device/chain/connector/account is hard to reach, make it reachable — that is the work, not an excuse to mock. If the existing tests here are shallow or mocked, fixing them is part of your change.
• Screenshots + logs at every phase, plus a complete walkthrough video/run-through of the entire feature or view, start to finish (bun run test:e2e:record).
• Manually review every artifact the change touches — never just the green check: client logs (console + network), server logs ([ClassName] …), the model trajectories in and out, before/after full-page screenshots, and the domain artifacts listed below for this package.
• No residuals. No shortcuts. The goal is not "done" — it is everything done. Clear every blocker by the hard path: build the real architecture, stand up the real model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a "follow-up." When unsure, research thoroughly, weigh the options, and ship the best, highest-effort, production-ready version. Keep going until every possibility is exhausted.

Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in <details>); attach each evidence type or explicitly mark it N/A with a reason — never leave it blank. If develop moved and changed behavior, re-capture evidence; stale proof is worse than none.

Capture & manually review for this package — native / on-device bridge:

• The capability run on a real device or simulator — not desktop Chromium against a mocked bridge (see #9967/#9580): device logs + the captured output (photo, OCR text, detection boxes, transcript, sensor reading).
• Parity vs the reference implementation where one exists (e.g. the Python/Ultralytics reference), with the numeric tolerances actually met.
• Permission-denied, no-hardware, and background/foreground lifecycle paths.
• A short recording of the on-device run; confirm the build under test is yours (versionName / a known on-screen change), not a stale install.