elizaos--eliza/packages/docs/ongoing-development/research/08-device-testing-pipeline.md

On-device testing pipeline (Android + iOS fleet)

Summary

The MVP is the LifeOps Personal Assistant (GitHub project 15): chat, onboarding, the current views, and LifeOps scheduling/reminders/goals/tasks, serving children, adults with ADHD/ADD/Asperger's/autism, neurotypical adults, and elderly users through real-life scenarios — no therapy language, no special rails. Every dev machine has Android and iOS devices plugged in; those devices ARE the mobile e2e environment (CI has no iOS e2e lane at all, and the Android CI lane is dispatch-only with a known x86_64 SIGSEGV limitation). The MVP question for this workstream: can any engineer or coding agent, on any dev machine, run one command per platform that builds current develop, installs it on the plugged-in device, runs the e2e lane, and hands back a triage-ready artifact bundle — and can they check at a glance whether every connected device runs develop HEAD?

Decision: do not build new infrastructure. Nearly every primitive already exists — per-platform orchestrators, a physical-iPhone deploy pipeline, a content-derived renderer build stamp carrying the git commit, capture scripts, an evidence-required gate. The work is (1) fixing a verified stale-install bug in the Android runner, (2) wiring the existing build stamp into a fleet-wide devices:status check, (3) collapsing scattered artifacts into one run-scoped bundle in inline-postable formats (MP4/JPG), and (4) chaining the already-proven physical-iPhone scripts into one command.

Current state

What exists today, verified in code:

Per-platform e2e orchestrators exist and are real (not mocks).

• packages/app/scripts/android-e2e.mjs:189-281 — single entrypoint: boots an emulator or reuses an attached device (lib/android-device.mjs:263-276 prefers the attached physical device), installs the APK, runs the on-device local-model chat smoke, drives every route on the real WebView via Playwright's Android driver, optional cloud probe. Loud non-zero exits.
• packages/app/scripts/ios-e2e.mjs:87-131 — simulator lane: boots a sim, builds by default (build:ios:local:sim), runs the auth deep-link smoke and the full-Bun on-device local chat smoke. Simulator only — there is no physical-iPhone e2e entrypoint; the pieces exist but are not chained (see below).

Verified bug: the Android runner runs stale builds by default. android-e2e.mjs:202-214: --build is opt-in, and the install step is if (!isInstalled(...)) — isInstalled (lib/android-device.mjs:382-392) only checks pm list packages. Consequences: (a) the default run executes whatever APK was installed last week; (b) even --build builds a fresh APK and then skips installing it when any version of the app is already on the device. This is the exact Capacitor bake-the-web-bundle-into-the-APK footgun the root CLAUDE.md warns about, live in the runner itself.

A build-identity stamp already exists — but is only enforced on the iOS simulator.

• packages/app/vite/renderer-build-manifest-plugin.ts:13-37 writes dist/eliza-renderer-build.json on every production renderer build: content-derived buildId, commit (git HEAD), variant, capacitorTarget, runtimeMode.
• packages/app/src/renderer-build-stamp.ts:1-36 exposes it at runtime as window.__ELIZA_RENDERER_BUILD__ and at the web root.
• packages/app/scripts/lib/ios-renderer-stamp.mjs:123-145 (assertInstalledIosAppRendererFresh) compares the installed sim app's stamp against fresh dist — used by mobile-local-chat-smoke.mjs:336 and ios-onboarding-smoke.mjs:184-218.
• packages/app-core/scripts/lib/mobile-lane-stamp.mjs:1-31 guards against wrong-lane dist reuse pre-Capacitor-sync (#11030).
• Gaps: no Android installed-freshness check anywhere (the stamp ships inside the APK at assets/public/eliza-renderer-build.json since Capacitor webDir: "dist", capacitor.config.ts:129, but nothing reads it back); no physical-iOS freshness check; no fleet-wide "what does each connected device run vs develop HEAD" report. ELIZAOS_VERSION_NAME/CODE stamping in run-mobile-build.mjs:807-887 only fires when the env vars are set — dev builds all report 1.0.

Physical-iPhone pipeline exists and is proven, but is four separate commands.

• packages/app/scripts/ios-device-deploy.mjs — unsigned build → provisioning-profile auto-discovery → graft → explicit nested signing → devicectl install → launch. --skip-appexes (lines 24-36, 429-441; PR #13174, merged 2026-07-04) strips the five app extensions so the main app deploys with only the app's own team profile — the unattended posture, since minting per-appex profiles needs an Xcode account session or an ASC API key, neither available headlessly on dev machines.
• packages/app/scripts/ios-device-capture.mjs — XCUITest BootCapture harness (screenshots via XCUIScreen, exports attachments + test-summary.json from the .xcresult, lines 522-537); retry-in-isolation for flake cascades landed in #13698.
• packages/app/scripts/ios-device-logs.mjs — bounded console capture + boot-trace pull (with the #11515 caveat: attached console SIGTRAPs the full-Bun engine, so engine observability uses --no-console --pull-boot-trace).
• No deploy ledger: nothing records which buildId/commit was last deployed to which physical device, and ios-device-deploy.mjs has no renderer-freshness assertion (verified: zero matches for renderer|stale|fresh in the file).

Capture and evidence tooling exists but targets the old evidence mechanics.

• packages/app/scripts/capture-ios-sim.mjs:48-79 records .mov via simctl io recordVideo (h264 in a QuickTime container) — GitHub does not render .mov inline in issues/PRs, so today's iOS captures cannot satisfy the new inline-evidence rule without a remux. capture-android-emu.mjs:84-158 records .mp4 (fine), plus logcat -d -t 500 and a backend-log pull.
• All capture defaults write to .github/issue-evidence/ (lib/issue-evidence.mjs:16-21) — the repo is moving to inline PR/issue posting (MP4 + JPG), so the committed-evidence directory stops being the destination.
• The green-with-nothing fix landed (#13624): resolveRequireEvidence + skip() exiting non-zero when evidence is required (lib/issue-evidence.mjs:60-100,246-260; sweep enforcement in scripts/e2e-recordings/run-all.mjs:56-61,388-402).

Failure surfacing is stdout-only and artifacts scatter across five locations.

• playwright.android.config.ts:36-43: reporters are list + html — no JUnit, no JSON; screenshots only-on-failure, trace retain-on-failure, landing in Playwright's own test-results/.
• iOS smokes throw with a screenshot path embedded in the error string (mobile-local-chat-smoke.mjs:1475-1492) — actionable only if you scroll the log.
• Artifact locations today: .github/issue-evidence/, e2e-recordings/<suite>/test-results/, packages/app/ios/build/{device-deploy-stage,device-logs,boot-capture}/, Playwright test-results/, reports/walkthrough/. No single per-run bundle.
• packages/app/scripts/walkthrough-device-matrix.mjs:20-26 is the closest thing to a fleet probe: it detects available platforms and writes an honest per-platform device-matrix.json (run | n/a + reason) — but it reports availability, not installed-build-vs-HEAD.

CI cannot replace the dev-machine fleet. .github/workflows/android-device-e2e.yml:1-35: workflow_dispatch only, plus a minimal PR slice behind the ci:device label; the file header documents that the on-device agent SIGSEGVs on stock x86_64 emulators, so the local-model route only greens on real arm64 hardware. There is no iOS e2e workflow at all (build-ios.yml is build-only on macos-15). The plugged-in device fleet is the only place the real mobile paths run.

No device lease. Concurrent agent sessions on one machine collide on the same simulator/device — ios-e2e.mjs:50-61 grabs the first booted simulator; nothing locks a serial/udid. This has caused real contention between concurrent sessions on this fleet.

Design considerations

• Wire, don't build. Every capability the mission asks for maps onto an existing script; the deltas are defaults, one new ~200-line status script, and bundle plumbing.
• Freshness must be stamp-driven, not rebuild-always. The renderer stamp is content-derived and carries the commit; comparing installed-vs-fresh buildId makes "skip the 4-minute rebuild when nothing changed" safe, and makes "reinstall when stale" automatic. Rebuild-always would make the runner too slow to be run habitually.
• Inline evidence formats are a capture-time concern. MP4 renders inline on GitHub; MOV does not. The simctl recording is already h264, so .mov → .mp4 is a container remux (ffmpeg -c copy), not a re-encode. Screenshots stay PNG for pixel assertions; the bundle emits JPG copies for posting.
• JUnit-ish summary is nearly free. Playwright ships junit and json reporters — adding them to playwright.android.config.ts is config. The non-Playwright iOS smokes need a small shared summary.json writer; ios-device-capture.mjs already writes test-summary.json, so the schema should extend that shape.
• Fail-fast doctrine applies to the runners. On a fleet machine with devices guaranteed plugged in, a runner that soft-skips "no device" is lying. The --require-evidence contract (#13624) already encodes this; the per-platform runners should arm it by default and keep the opt-out.
• Point-of-failure forensics beat post-hoc archaeology. The runner already owns the child-process boundary (android-e2e.mjs:162-171 run()); catching a failed step there and immediately capturing screenshot + logcat -d / boot-trace is a ~30-line change per platform, not a framework.

Open questions → answers

Q1. Should the per-platform runner always rebuild? No — stamp-driven. Default flow: compute fresh-dist buildId (building only if dist is missing/stale for the lane per mobile-lane-stamp.mjs), read the installed stamp, and rebuild+reinstall only on mismatch. --skip-build stays for tight iteration, --force-build for paranoia. Rationale: correctness of the Capacitor footgun without making the habitual command cost 5+ minutes when nothing changed.

Q2. Where do triage bundles live now that .github/issue-evidence/ is deprecated as a destination? A gitignored run-scoped directory: packages/app/device-e2e-output/<platform>-<runId>/ with inline/ (the MP4s + JPGs sized for posting), logs/, summary.json, and junit.xml. The runner prints the absolute path as its last line so an agent can post inline/ contents directly to the PR. Existing .github/issue-evidence/ writers remain untouched for old flows but the runners stop defaulting there.

Q3. JUnit or JSON as the machine-readable summary? Both, JSON canonical. summary.json (shared schema: lane, device identity, installed buildId/commit, steps[] with status/duration/artifact paths) is what coding agents parse; junit.xml is a free byproduct of the Playwright reporter for anything expecting JUnit. Undecidable-free: nothing consumes JUnit today, so JSON-canonical costs nothing.

Q4. Do we try to green the CI emulator's local-model route for MVP? No. The x86_64 SIGSEGV is a hardware-emulation issue documented in android-device-e2e.yml; the fix (self-hosted arm64 runner) is infrastructure the MVP does not need while every dev machine has real arm64 devices attached. CI stays dispatch/label-gated; the fleet is the mobile e2e environment of record.

Q5. Do we need a device farm / fleet service? No. One machine, a handful of devices, multiple concurrent agent sessions — the failure mode is contention, not scale. A lease lockfile keyed by serial/udid under the state dir (stale-expiry, holder PID + session id) is sufficient and deletes nothing if unused.

Q6. What about the iOS appex signing blocker for unattended runs? Accept --skip-appexes as the default unattended posture (already merged, PR #13174) and have the runner log loudly that widgets/keyboard/device-activity surfaces are untested on device. Minting per-appex profiles requires an ASC API key — a real owner decision (credential provisioning), default: park it; the MVP surfaces (chat, onboarding, views, LifeOps) do not live in appexes.

Q7. How does a physical iPhone report its installed build when we can't read its app container? Deploy-time ledger. ios-device-deploy.mjs stages the App.app locally, so it can read the staged bundle's renderer stamp and append {udid, buildId, commit, deployedAt} to a state-dir ledger; devices:status reads the ledger and reports "unknown — no ledger entry" honestly when a device was flashed by other means. The boot-trace pull (ios:device:logs --pull-boot-trace) remains the ground-truth cross-check.

Recommendation

Minimal-scope MVP plan, in order:

1. Fix the Android stale-install bug and make freshness stamp-driven in android-e2e.mjs (P0). Add the Android installed-stamp reader (pm path → pull APK → read assets/public/eliza-renderer-build.json) to lib/android-device.mjs; reinstall on mismatch; make --build actually reinstall.
2. Ship devices:status (P0): one read-only script enumerating adb devices, simctl list, devicectl list devices; per device print installed buildId/commit vs origin/develop HEAD and fresh dist; exit non-zero on --require-fresh.
3. Triage bundle + machine-readable summary (P1): run-scoped output dir, Playwright junit+json reporters, shared summary.json writer, MOV→MP4 remux + JPG copies in inline/.
4. One-command physical-iPhone lane (P1): chain deploy (--skip-appexes default) → local-chat smoke → BootCapture → logs → bundle, and write the deploy ledger.
5. Point-of-failure forensics (P1): failed step ⇒ immediate screenshot + device log snapshot into the bundle; runner exit summary names the failing step + artifact paths.
6. Device lease lock (P2): serial/udid-keyed lockfile so concurrent sessions queue instead of colliding.

Out of scope (MVP non-goals)

• Greening the CI x86_64 emulator local-model route or standing up self-hosted arm64 runners.
• Automating per-appex provisioning-profile minting (needs ASC API key — owner decision).
• Any device-farm service, remote-device brokering, or multi-machine fleet orchestration.
• Changes to desktop capture lanes (capture-linux-desktop.mjs, capture-windows-desktop.mjs, capture-macos-desktop.mjs) — they already exist and are not on the mobile-fleet path.
• New mobile test content (LifeOps scenario coverage is the scenarios workstream; this workstream is the pipeline those tests run on).
• Store-release signing/TestFlight automation (apple-store-release.yml, android-release.yml are separate release lanes).

Proposed issues

1. [device-testing] android-e2e runs stale builds: install-if-missing skips fresh APKs — make freshness stamp-driven (P0)
2. [device-testing] devices:status — report installed buildId/commit per connected device vs develop HEAD (P0)
3. [device-testing] Triage-ready artifact bundle per device-e2e run: MP4 + JPG + logs + summary.json + junit.xml (P1)
4. [device-testing] One-command physical-iPhone e2e lane (deploy → smoke → capture → bundle) with deploy ledger (P1)
5. [device-testing] Point-of-failure forensics: auto-capture screenshot + device logs when a runner step fails (P1)
6. [device-testing] Device lease lock: stop concurrent agent sessions colliding on one simulator/device (P2)