chore: import upstream snapshot with attribution
Benchmark Bridge Tests / benchmark (bunx @biomejs/biome check packages/lifeops-bench/src, benchmark-lint) (push) Waiting to run
Benchmark Bridge Tests / benchmark (bunx vitest run --config packages/lifeops-bench/vitest.config.ts --root packages/lifeops-bench --passWithNoTests, benchmark-tests) (push) Waiting to run
Build Agent Image / build-and-push (push) Waiting to run
Chat shell gestures / Chat shell gesture + parity e2e (push) Waiting to run
ci / test (push) Waiting to run
ci / lint-and-format (push) Waiting to run
ci / build (push) Waiting to run
ci / dev-startup (push) Waiting to run
Cloud Gateway Discord / Test (push) Waiting to run
Cloud Gateway Webhook / Test (push) Waiting to run
Cloud Tests / lint-and-types (push) Waiting to run
Cloud Tests / unit-tests (push) Waiting to run
Cloud Tests / integration-tests (push) Waiting to run
Cloud Tests / e2e-tests (push) Blocked by required conditions
CodeQL Advanced / Analyze (javascript-typescript) (push) Waiting to run
Deploy Apps Worker (Product 2) / Determine environment (push) Waiting to run
Deploy Apps Worker (Product 2) / Deploy apps worker to apps-control host (${{ needs.determine-env.outputs.environment }}) (push) Blocked by required conditions
Deploy Eliza Provisioning Worker / Determine environment (push) Waiting to run
Deploy Eliza Provisioning Worker / Deploy worker to Hetzner host (${{ needs.determine-env.outputs.environment }} @ ${{ needs.determine-env.outputs.deployment_sha }}) (push) Blocked by required conditions
Dev Smoke / Classify changed paths (push) Waiting to run
Dev Smoke / bun run dev onboarding chat (push) Blocked by required conditions
Dev Smoke / Vite HMR dependency-level smoke (push) Blocked by required conditions
Electrobun Submodule Guard / electrobun gitlink is fetchable (push) Waiting to run
gitleaks / gitleaks (push) Waiting to run
Markdown Links / Relative Markdown Links (push) Waiting to run
Publish @elizaos/example-code / check_npm (push) Waiting to run
Publish @elizaos/example-code / publish_npm (push) Blocked by required conditions
Publish @elizaos/plugin-elizacloud / verify_version (push) Waiting to run
Publish @elizaos/plugin-elizacloud / publish_npm (push) Blocked by required conditions
Quality (Extended) / Homepage Build (PR smoke) (push) Waiting to run
Quality (Extended) / Comment-only diff guard (push) Waiting to run
Quality (Extended) / Format + Type Safety Ratchet (push) Waiting to run
Quality (Extended) / Develop Gate (secret scan + UI determinism) (push) Waiting to run
Quality (Extended) / Develop Gate (lint) (push) Waiting to run
Sandbox Live Smoke / Sandbox live smoke (push) Waiting to run
Snap Build & Test / Build Snap (amd64) (push) Waiting to run
Snap Build & Test / Build Snap (arm64) (push) Waiting to run
supply-chain / sbom (push) Waiting to run
supply-chain / vulnerability-scan (push) Waiting to run
Build, Push & Deploy to Phala Cloud / build-and-push (push) Waiting to run
Test Packaging / Validate Packaging Configs (push) Waiting to run
Test Packaging / PyPI on Python ${{ matrix.python }} (push) Waiting to run
Test Packaging / Pack & Test JS Tarballs (push) Waiting to run
Test Packaging / elizaos CLI global-install smoke (node + bun) (push) Waiting to run
UI Fixture E2E / ui-fixture-e2e (push) Waiting to run
UI Fixture E2E / fixture-e2e (push) Waiting to run
UI Story Gate / story-gate (push) Waiting to run
vault-ci / test (macos-latest) (push) Waiting to run
vault-ci / test (ubuntu-latest) (push) Waiting to run
vault-ci / test (windows-latest) (push) Waiting to run
vault-ci / app-core wiring tests (push) Waiting to run
verify-patches / verify patches/CHECKSUMS.sha256 (push) Waiting to run
Voice Benchmark Smoke / voice-emotion fixture smoke (push) Waiting to run
Voice Benchmark Smoke / voiceagentbench fixture smoke (push) Waiting to run
Voice Benchmark Smoke / voicebench-quality unit smoke (push) Waiting to run
Voice Benchmark Smoke / voicebench TypeScript unit (no audio) (push) Waiting to run
Voice Benchmark Smoke / voice bench smoke summary (push) Blocked by required conditions
Windows CI / windows ([bun run --cwd packages/app-core test bun run --cwd packages/elizaos test bun run --cwd packages/cloud/shared test], app-and-cli) (push) Waiting to run
Windows CI / windows ([bun run --cwd packages/scenario-runner test bun run --cwd packages/vault test bun run --cwd packages/security test bun run --cwd plugins/plugin-coding-tools test], framework-packages) (push) Waiting to run
Windows CI / windows ([bun run --cwd plugins/plugin-elizacloud test bun run --cwd plugins/plugin-discord test bun run --cwd plugins/plugin-anthropic test bun run --cwd plugins/plugin-openai test bun run --cwd plugins/plugin-app-control test bun run --cwd plugins/pl… (push) Waiting to run
Windows CI / windows ([node packages/scripts/run-turbo.mjs run build --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/agent --concurrency=4 node packages/scripts/run-bash-linux-only.mjs scripts/verify-riscv64-buildpaths.sh node packages/scripts/run… (push) Waiting to run
Windows CI / windows ([node packages/scripts/run-turbo.mjs run typecheck --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/cloud-shared --concurrency=4 bun run --cwd packages/core test bun run --cwd packages/shared test], core-runtime, 75) (push) Waiting to run
Test Packaging / Build & Test PyPI Package (push) Waiting to run
Voice Workbench / headless workbench (mocked backends) (push) Has been cancelled
Voice Workbench / real acoustic lane (nightly, provisioned only) (push) Has been cancelled

This commit is contained in:
wehub-resource-sync
2026-07-13 12:43:05 +08:00
commit 426e9eeabd
41828 changed files with 9656266 additions and 0 deletions
+68
View File
@@ -0,0 +1,68 @@
# CODEOWNERS — elizaOS monorepo
#
# Required reviewers for sensitive areas. Used by GitHub branch-protection
# rules to enforce change management (SOC2 CC8.1 / CC6.1).
#
# IMPORTANT — operator action required:
# The team handles below (`@elizaos/<role>`) are PLACEHOLDERS. Before enabling
# branch protection with "Require review from Code Owners", replace each
# handle with a real GitHub team in the elizaos org. The format is
# `@org-name/team-slug`. Verify each team exists with
# `gh api orgs/elizaos/teams/<slug>`.
#
# Order matters: later rules override earlier ones (last match wins).
# Default — at least one core maintainer.
* @elizaos/core-maintainers
# CI / CD pipeline definitions and reusable actions.
/.github/ @elizaos/security-team @elizaos/platform-team
/.github/workflows/ @elizaos/security-team
/.github/actions/ @elizaos/security-team
/.github/dependabot.yml @elizaos/security-team
# Cloud / backend — billing, auth, hosted APIs.
/packages/cloud/api/ @elizaos/backend-leads
/packages/cloud/shared/ @elizaos/backend-leads
/packages/cloud/services/ @elizaos/backend-leads
/packages/cloud/infra/ @elizaos/backend-leads @elizaos/platform-team
/packages/cloud/routing/ @elizaos/backend-leads
/packages/cloud/sdk/ @elizaos/backend-leads
# Security and crypto-handling code — mandatory security review.
/packages/security/ @elizaos/security-team
/packages/vault/ @elizaos/security-team
/packages/contracts/ @elizaos/security-team @elizaos/backend-leads
# Plugin runtime / sandboxing surface.
# The host-shim cluster (base + android/ios/electrobun) now lives under
# plugin-remote-manifest/src/host-shim/ (subpath exports).
/packages/plugin-worker-runtime/ @elizaos/runtime-team
/packages/plugin-sub-agent-claude-code/ @elizaos/runtime-team @elizaos/security-team
/packages/plugin-remote-manifest/ @elizaos/runtime-team @elizaos/security-team
# ML / training / hardware.
/packages/training/ @elizaos/ml-hw-lead
/packages/feed/ @elizaos/ml-hw-lead
# Deployment artifacts.
/deploy/ @elizaos/platform-team
# Install / bootstrap surface.
/install/ @elizaos/security-team
# Dependency patches.
/patches/ @elizaos/backend-leads @elizaos/security-team
/plugins/**/patches/ @elizaos/backend-leads @elizaos/security-team
# Test-gating baselines and deny-lists (#10104 Tier-3) — these files are the
# knobs that loosen CI: raising a ratchet baseline budgets more debt, and
# adding a spec to the deny-list silently removes it from PR coverage. A
# feature PR must not adjust them without an explicit owner review.
/packages/scripts/type-safety-ratchet-baseline.json @elizaos/security-team @elizaos/platform-team
/packages/scripts/type-duplication-audit.baseline.json @elizaos/security-team @elizaos/platform-team
/packages/scripts/ui-determinism-baseline.json @elizaos/security-team @elizaos/platform-team
/packages/app/test/ui-smoke/.pr-deny-list.json @elizaos/security-team @elizaos/platform-team
# Documentation only — lighter review.
/docs/security/ @elizaos/security-team
+109
View File
@@ -0,0 +1,109 @@
# Agent Fleet — coordination guide
Canonical rules for every AI agent (and human) working this repo as part of the
fleet. Code conventions live in `AGENTS.md`/`CLAUDE.md`; the evidence bar lives
in `CONTRIBUTING.md`. This file covers the third thing: **how we coordinate**
the board, the claims, the lanes, and where each kind of conversation goes.
## Where things live
| Surface | Used for | Not for |
|---|---|---|
| **GitHub Project 12** (org board) | Live work state: every card is an issue, columns below | Chat, design debate |
| **Issues** | Actionable cards only — one bug/feature/task each, `launch-qa` labels | Status chatter (that's the Discussion) |
| **Discussions → "Agent Fleet — Coordination HQ"** (pinned, General) | Standups, claims visibility, cross-lane questions, incident threads | Anything needing a card (open an issue) |
| **PRs** | The work itself + its evidence | Coordination side-talk |
| **`.github/FLEET.md`** (this file) | The rules — PR a change to propose a rule change | — |
Long-running "tracking issues" with hundreds of linear comments are
**deprecated** for coordination: threads in the Discussion replace them. Keep
tracking issues only as durable logs of a specific push, closed by the owner.
## Identity
- Every agent has a **lane tag**`[qa-agent]`, `[maintainer]`,
`[cloud-agent]`, `[core-brain]`, … — and **signs every comment** with it.
- One lane tag = one running context. If you inherit a lane, say so in the
Discussion before acting in it.
## The loop
1. **Read first**: the Discussion's latest posts + the board. Never start work
someone has claimed.
2. **Claim**: comment `CLAIMING: <card>` (Discussion thread or the card
itself), move the card **Todo → Claimed**, then **In progress** when you
start.
3. **Work to the evidence bar** (`CONTRIBUTING.md`): real end-to-end proof, no
mocks, no shortcuts.
4. **Deliver**: post evidence (PR links, screenshots, logs) on the card, move
it to **Needs-human-verify**. Only the owner (nubs) moves cards to **Done**
or closes the push.
5. **Next card immediately.** Blocked >30 min → say so where you claimed, move
the card back, pick another.
Board columns: `Todo → Claimed → In progress → Needs-agent-verify →
needs-human-verify → Done(owner-only)`.
## Shared levers — claim before you mutate
Some things exist once. Announce **before** touching, wait for objections when
another lane is active on it, release when done:
- Prod deploys (Workers, Pages) and the develop→main promote
- Staging environment (deploys, env vars, secrets, seeded data)
- The production database (any write — even additive grants get a
transparency note with the ledger row id)
- DNS / Cloudflare settings / repo settings & rulesets
- **Physical devices** (the Seeker phone + its adb) — one installer at a time
- Worker secrets, CI runner capacity
Case study: two agents built + installed on the Seeker in parallel
(2026-07-05). Cost: a duplicated build and a mid-air de-conflict. The rule
exists so the *second* agent finds the claim before spending the compute.
## Merge & deploy etiquette
- **Never self-merge your own PR** into a lane that has a reviewer
(maintainer shepherds merges). Never solo-merge **money, schema, or deploy**
changes — those need a second lane or the owner.
- **Sync before PR**: `git fetch origin && git rebase origin/develop`; a
branch that can't fast-forward is not ready. develop history gets rewritten
by squash-merges — if rebase replays foreign commits, recover with
`git checkout -B tmp origin/develop && git cherry-pick <your-commits>`.
- **Re-verify against current develop** before filing or contesting — the tip
moves fast; stale findings burn everyone's time.
- Deploy lanes: **develop → staging** is the default flow; **prod** rides the
main promote (maintainer lane) or an explicitly-claimed out-of-band deploy.
- Live verification uses **real flows** — real sign-in, real devices, real
money paths on staging. Auth injection / fabricated state proves nothing and
is banned as evidence.
## Using the Discussion HQ (features, not flat chat)
The HQ is **[#14308 — Agent Fleet Coordination HQ](https://github.com/orgs/elizaOS/discussions/14308)**.
Use the platform's features — a flat comment stream doesn't scale to a fleet:
- **One claim = one thread.** Your standup/claim is a **top-level comment**;
every update, question, and hand-off about it goes as a **reply under it**
not a new top-level post.
- **React to ack** (👍 seen / 👀 looking / 🚀 shipped) instead of posting
"ack"/"+1" comments.
- **Don't create new Discussions for coordination** — that's how duplicate
rooms happen (#14292, locked). Per-topic **workstream discussions** (e.g.
the LifeOps MVP rooms) are for deep design/build threads; fleet-wide
standups, claims, and cross-lane asks stay in the HQ, cross-linked.
- **Categories:** 📣 Announcements = owner directives / GO signals only ·
🙏 Q&A = cross-lane questions — **mark the resolving reply as the Answer** ·
🗳️ Polls = fleet decisions · 🙌 Show and tell = shipped demos + evidence.
- **Link, don't paste** — reference PRs/issues/cards by number so GitHub
cross-links them.
## Communication hygiene
- Sign everything. Link evidence, not adjectives.
- Read the last hour of the Discussion before posting — don't duplicate.
- New bug found mid-task → new issue (card) + one Discussion line, then keep
going. Don't bury findings in comment threads.
- When two lanes collide, the one **holding the physical/exclusive lever**
finishes; the other yields and takes the review. Escalate to the owner only
on real deadlock.
+42
View File
@@ -0,0 +1,42 @@
---
name: Agent work item
about: Create a scoped issue that can be added to a GitHub Project kanban
title: "[agent] "
labels: "Agent"
assignees: ""
---
## Project / Coordination
- Project board:
- Parent tracker or epic:
- Coordination Discussion:
- Suggested lane tag:
## Scope
<!-- Describe the smallest verifiable unit of work. -->
## Acceptance Criteria
- [ ] Acceptance criterion:
## Evidence Plan
Keep every applicable row visible. Use `N/A - <reason>` only when a row truly
does not apply.
- [ ] Screenshots or recording:
- [ ] Logs:
- [ ] Real-LLM trajectories:
- [ ] Domain artifacts:
- [ ] CI or local commands:
## Claim / Status Notes
Before working, comment `CLAIMING: <scope>` with your lane tag, add this issue
to the active Project, set `Status = Claimed`, and set `Claimed by = <tag>`.
When active work starts, move the card to `In progress`.
Follow the full workflow in [`CONTRIBUTING.md`](../../CONTRIBUTING.md) (claim
protocol, GitHub Projects states, and the mechanically-enforced evidence bar).
+42
View File
@@ -0,0 +1,42 @@
---
name: Bug report
about: Create a report to help us improve
title: ""
labels: "bug"
assignees: ""
---
**Describe the bug**
<!-- A clear and concise description of what the bug is. -->
**To Reproduce**
<!-- Steps to reproduce the behavior. -->
**Expected behavior**
<!-- A clear and concise description of what you expected to happen. -->
**Screenshots / recording of the wrong behavior (required for anything visible)**
<!-- Attach JPG screenshots and/or an MP4 recording of the broken behavior
inline here. Videos must be MP4 so GitHub renders them; prefer JPG over
PNG for screenshots. A visible bug without a screenshot/recording of the
wrong behavior is not actionable. -->
**Evidence / reproduction proof**
Attach proof inline in this issue (drag-and-drop) that lets a maintainer
reproduce and inspect the real failure:
- [ ] MP4 recording or JPG screenshots of the broken behavior.
- [ ] Backend logs (`[ClassName] ...`) and frontend console/network logs when relevant (wrap long output in a `<details>` block).
- [ ] Real-LLM trajectory when the bug involves agent/action/prompt/model behavior.
- [ ] Domain artifacts when relevant (DB rows, memories, scheduled tasks, generated files, wallet/on-chain output).
If an item is unavailable, keep the row visible and write `N/A - <reason>`.
**Additional context**
<!-- Add any other context about the problem here. -->
+14
View File
@@ -0,0 +1,14 @@
blank_issues_enabled: false
contact_links:
- name: LifeOps Personal Assistant MVP board
url: https://github.com/orgs/elizaOS/projects/15
about: The active kanban. MVP work items live here; workstream design docs are under packages/docs/ongoing-development/.
- name: Security vulnerability
url: https://github.com/elizaOS/eliza/blob/develop/SECURITY.md
about: Do not open a public issue for security bugs. Read the security policy for private reporting instructions.
- name: Discord community
url: https://discord.gg/ai16z
about: Chat with the team and other contributors (see the development-feed channel).
- name: Documentation
url: https://docs.elizaos.ai/
about: Guides, API reference, and plugin documentation.
+49
View File
@@ -0,0 +1,49 @@
---
name: Epic / tracking issue
about: Track a multi-PR project with acceptance criteria and evidence
title: "[Epic] "
labels: "documentation"
assignees: ""
---
## Goal
<!-- What user-visible or maintainer-visible state should be true when this epic is done? -->
## Design doc
<!-- Link the design doc under packages/docs/ongoing-development/ that this
epic implements. Non-trivial epics should have one; write it first. -->
## Scope
<!-- List the product areas, packages, services, or workflows included. -->
## Non-goals
<!-- Explicitly list related work that is out of scope. -->
## Work Items
- [ ] Item 1
- [ ] Item 2
## Acceptance Criteria
- [ ] The implemented behavior is verified end to end.
- [ ] Required tests and CI gates are added or updated.
- [ ] Evidence is posted inline in this epic and/or the child PRs (MP4 video, JPG screenshots, logs) — not committed to the repo.
## Evidence Plan
<!-- Keep every applicable row visible. Use `N/A - <reason>` only when a row truly does not apply. -->
- [ ] Screenshots or recording:
- [ ] Logs:
- [ ] Real-LLM trajectories:
- [ ] Domain artifacts:
- [ ] CI runs:
## Dependencies / Sequencing
<!-- Link prerequisite issues or PRs. -->
+43
View File
@@ -0,0 +1,43 @@
---
name: Feature request
about: Suggest an idea for this project
title: ""
labels: "enhancement"
assignees: ""
---
**Is your feature request related to a problem? Please describe.**
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
**Describe the solution you'd like**
<!-- A clear and concise description of what you want to happen. -->
**Describe alternatives you've considered**
<!-- A clear and concise description of any alternative solutions or features you've considered. -->
**Design doc / board**
<!-- Link the relevant design doc under packages/docs/ongoing-development/ if
one exists. If this is MVP work, add the issue to the LifeOps Personal
Assistant MVP board: https://github.com/orgs/elizaOS/projects/15 -->
**Additional context**
<!-- Add any other context or current-state screenshots (JPG) about the feature request here. -->
**Evidence / expected proof for implementation**
For UI or user-facing features, attach current-state JPG screenshots or a short
MP4 recording that shows the workflow today. The implementing PR must include,
posted inline in the PR:
- [ ] Before and after full-page screenshots for affected UI surfaces (desktop and mobile), as JPG.
- [ ] An MP4 video walkthrough of the full flow (MP4 renders inline in GitHub).
- [ ] Backend logs and frontend console/network logs when a real code path is involved.
- [ ] Real-LLM trajectory when the feature changes agent/action/prompt/model behavior.
- [ ] Domain artifacts when relevant (DB rows, memories, scheduled tasks, generated files, wallet/on-chain output).
If an item is unavailable, keep the row visible and write `N/A - <reason>`.
+45
View File
@@ -0,0 +1,45 @@
# actionlint configuration
# https://github.com/rhysd/actionlint/blob/main/docs/config.md
# Most jobs run on GitHub-hosted runners (ubuntu-24.04 / ubuntu-24.04-arm /
# windows-2025 / macos-15 etc.). The labels below are self-hosted runner
# labels we use:
# kvm — Linux x86_64 host with /dev/kvm available, used for Cuttlefish/AOSP
# builds (.github/workflows/elizaos-cuttlefish.yml).
# gpu-cuda-12.6 — Linux x86_64 host with an NVIDIA GPU + CUDA 12.6 driver
# stack, used for the eliza-1 training stack QJL nvcc
# build (.github/workflows/training-stack.yml).
# eliza — Linux x86_64 provisioned Eliza self-hosted pool, used for real
# voice workflow_dispatch/nightly lanes that must run on repo-owned
# hardware and fail hard when model/secret prerequisites are absent.
self-hosted-runner:
labels:
- kvm
- gpu-cuda-12.6
- eliza
paths:
.github/workflows/*.yml:
ignore:
# Keep workflow lint focused on actionable failures. actionlint surfaces
# shellcheck `info` and `style` diagnostics as errors by default, which
# makes this repo's workflow lint fail without any workflow syntax,
# expression, or shell warning/error findings.
- "shellcheck reported issue in this script: SC[0-9]+:(info|style):"
.github/workflows/*.yaml:
ignore:
- "shellcheck reported issue in this script: SC[0-9]+:(info|style):"
.github/workflows/release.yml:
ignore:
# Optional signing secrets — linter cannot verify repo secrets exist
- "Context access might be invalid:.+"
.github/workflows/release-electrobun.yml:
ignore:
- "Context access might be invalid:.+"
.github/workflows/android-release.yml:
ignore:
- "Context access might be invalid:.+"
.github/workflows/apple-store-release.yml:
ignore:
- "Context access might be invalid:.+"
@@ -0,0 +1,112 @@
name: Setup Cloud Test Environment
description: Common setup for Bun-based cloud test jobs with optional local services. Runs from packages/cloud/shared/.
inputs:
bun-version:
description: Bun version to install (pinned to avoid GitHub API rate-limit flakes).
required: false
# pinned: floating canary hangs `bun install` on hosted CI runners and
# writes lockfileVersion 2 which breaks --frozen-lockfile (#11184/#9454).
# Cloud Tests + cloud-e2e call this action without an override, so this
# default is what actually runs on those (queue-starved) lanes.
default: "1.3.14"
setup-db:
description: Start PGlite TCP server and run migrations.
required: false
default: "false"
db-backend:
description: "Database backend to start when setup-db is true: postgres or pglite."
required: false
default: "postgres"
runs:
using: composite
steps:
- uses: actions/setup-node@v6
with:
node-version: "22"
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2
with:
bun-version: ${{ inputs.bun-version }}
- name: Cache Bun install
uses: actions/cache@v5
with:
path: ~/.bun/install/cache
key: bun-${{ runner.os }}-${{ hashFiles('packages/cloud/shared/bun.lock') }}
restore-keys: |
bun-${{ runner.os }}-
- name: Install dependencies
shell: bash
working-directory: packages/cloud/shared
run: |
for attempt in 1 2 3; do
if [ "$attempt" -eq 3 ]; then
bun install --no-save --verbose && git diff --exit-code -- bun.lock && exit 0
else
bun install --no-save && git diff --exit-code -- bun.lock && exit 0
fi
echo "bun install attempt $attempt failed or changed bun.lock; retrying in 10s..."
sleep 10
done
exit 1
- name: Build linked elizaOS workspaces
shell: bash
working-directory: packages/cloud/shared
run: bun run build:linked-workspaces
- name: Validate migration journal
shell: bash
working-directory: packages/cloud/shared
run: bun run db:check-migrations
- name: Start PostgreSQL test server
if: inputs.setup-db == 'true' && inputs.db-backend == 'postgres'
shell: bash
run: |
docker rm -f eliza-cloud-test-postgres >/dev/null 2>&1 || true
docker run \
--name eliza-cloud-test-postgres \
-e POSTGRES_HOST_AUTH_METHOD=trust \
-e POSTGRES_DB=postgres \
-p 5432:5432 \
-d pgvector/pgvector:pg16
timeout 60 bash -c 'until docker exec eliza-cloud-test-postgres pg_isready -U postgres -d postgres; do sleep 0.5; done'
for attempt in {1..120}; do
if bun -e 'import pg from "pg"; const { Client } = pg; const client = new Client({ connectionString: process.env.DATABASE_URL }); await client.connect(); await client.query("select 1"); await client.end();' >/dev/null 2>&1; then
exit 0
fi
sleep 0.5
done
docker logs eliza-cloud-test-postgres
exit 1
- name: Start PGlite TCP server
if: inputs.setup-db == 'true' && inputs.db-backend == 'pglite'
shell: bash
working-directory: packages/cloud/shared
env:
PGLITE_IN_MEMORY: "1"
PGLITE_PORT: "5432"
PGLITE_HOST: "127.0.0.1"
PGLITE_MAX_CONNECTIONS: "64"
run: |
bun run pglite:server &
timeout 30 bash -c 'until bash -c "echo >/dev/tcp/127.0.0.1/5432" 2>/dev/null; do sleep 0.5; done'
- name: Run database migrations
if: inputs.setup-db == 'true'
shell: bash
working-directory: packages/cloud/shared
env:
DATABASE_URL: postgresql://postgres@127.0.0.1:5432/postgres
TEST_DATABASE_URL: postgresql://postgres@127.0.0.1:5432/postgres
run: bun run db:migrate
@@ -0,0 +1,20 @@
name: Normalize Snapd Root Ownership
description: Best-effort repair of root ownership on ephemeral runners before invoking snapd-based build steps. Soft-fails when sudo is unavailable so downstream snapcraft errors remain the authoritative signal.
runs:
using: composite
steps:
- shell: bash
run: |
set -euo pipefail
ROOT_OWNER="$(stat -c '%u:%g' /)"
echo "Runner root owner: ${ROOT_OWNER}"
if [ "${ROOT_OWNER}" = "0:0" ]; then
echo "Root is already 0:0 — snapd canonicalization OK"
exit 0
fi
echo "Attempting to repair / ownership so snapd/systemd path canonicalization accepts this runner"
if sudo -n chown root:root / 2>/dev/null; then
echo "Chown succeeded; new owner: $(stat -c '%u:%g' /)"
else
echo "::warning::Unable to chown / to root:root (sudo unavailable or passwordless sudo disabled). snapd may fail downstream. Remediation: run this job on a standard ubuntu-latest / ubuntu-24.04 GitHub-hosted runner, or configure the container image so / is owned by 0:0."
fi
@@ -0,0 +1,404 @@
name: "Setup Bun Workspace"
description: "Install Bun workspace dependencies and required native deps for eliza CI jobs."
inputs:
bun-version:
description: "Bun version to install"
required: false
default: "canary"
python-version:
description: "Python version for node-gyp fallbacks"
required: false
default: "3.12"
setup-python:
description: "Whether to set up Python for node-gyp fallbacks"
required: false
default: "true"
install-protoc:
description: "Whether to install protoc for Rust plugin builds"
required: false
default: "true"
force-protoc-github-fallback:
description: "Force the pinned GitHub protoc fallback path for setup action self-tests"
required: false
default: "false"
install-command:
description: "Dependency install command"
required: false
default: "bun install --frozen-lockfile --ignore-scripts"
install-native-deps:
description: "Whether to install native apt dependencies"
required: false
default: "true"
run-postinstall:
description: "Whether to run the repo postinstall step explicitly"
required: false
default: "true"
skip-avatar-clone:
description: "Set SKIP_AVATAR_CLONE=1 for postinstall"
required: false
default: "false"
no-vision-deps:
description: "Set ELIZA_NO_VISION_DEPS=1 for postinstall"
required: false
default: "false"
init-submodules:
description: "Run `git submodule update --init --recursive` before install"
required: false
default: "false"
setup-node:
description: >-
Whether to set up Node.js. Self-hosted runners are not guaranteed to have
`node` on PATH, and this action's very first step (turbo-cache-key) shells
out to `node`, so leave this on unless the caller already ran
actions/setup-node in the same job.
required: false
default: "true"
node-version:
description: "Node.js version to set up when setup-node is true"
required: false
default: "24"
runs:
using: "composite"
steps:
# Self-hosted (hetzner-robot) runners are not guaranteed to expose `node`
# on the step PATH, and the very next step (turbo-cache-github ->
# turbo-cache-key.mjs) invokes `node` directly. Without this, jobs that
# rely solely on this composite for their toolchain (coverage-gate,
# feed-env-audit, scenario-matrix, voice-workbench) fail with
# "node: command not found" on runners missing a system Node. setup-node is
# idempotent, so callers that already ran it (keyless-harness, the audits)
# just re-select the same version. Toggle off via setup-node: "false".
- name: Setup Node.js
if: ${{ inputs.setup-node == 'true' }}
# actions/setup-node@v4 (48b55a0)
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ inputs.node-version }}
# Persist Turbo's local filesystem cache (.turbo) through the pinned
# GitHub-native shim (#12341) — the single Turbo cache regime, replacing the
# Vercel SaaS remote cache. The shim keys on the deterministic
# turbo-cache-key hash with no github.job fragment, so one entry is shared
# across every job instead of the ~30-way fragmentation the old per-job key
# caused; Turbo re-validates task hashes on restore, so a cross-job hit is
# always safe. This keeps the full 14-way typecheck off the runner on warm
# caches (cold ran it OOM/SIGTERM). No SaaS, no secrets; a miss is non-fatal.
- name: Restore Turbo cache (GitHub-native)
uses: ./.github/actions/turbo-cache-github
# actions/setup-python@v6 downloads a CPython build requiring the
# x86-64-v3 CPU baseline; some [self-hosted, hetzner-robot] runners expose
# a lower ISA and the download aborts with "CPU ISA level is lower than
# required" (exit 127), killing the whole step even though Python here is
# only a best-effort node-gyp fallback (#14188). GitHub-hosted runners are
# never affected, so this probe (and the fallback it enables) only runs on
# self-hosted runners via runner.environment — hosted CI takes the
# untouched actions/setup-python@v6 path below.
- name: Detect usable system Python (self-hosted ISA guard)
id: system-python
if: ${{ inputs.setup-python == 'true' && runner.environment == 'self-hosted' }}
shell: bash
run: |
if command -v python3 >/dev/null 2>&1 && python3 -c "import sys" >/dev/null 2>&1; then
echo "python-bin=$(command -v python3)" >> "$GITHUB_OUTPUT"
fi
- name: Setup Python
if: ${{ inputs.setup-python == 'true' && steps.system-python.outputs.python-bin == '' }}
# Skipped outright when the probe above found a usable system Python.
# If it didn't (a self-hosted runner with no system python3 either),
# don't let an ISA-incompatible download turn into a hard job failure —
# hosted runners are never in this branch and keep fail-fast behavior.
continue-on-error: ${{ runner.environment == 'self-hosted' }}
uses: actions/setup-python@v6
with:
python-version: ${{ inputs.python-version }}
- name: Install setuptools for distutils (Python 3.12+)
if: ${{ inputs.setup-python == 'true' }}
shell: bash
env:
RUNNER_ENVIRONMENT: ${{ runner.environment }}
run: |
python_bin="${{ steps.system-python.outputs.python-bin }}"
if [ -z "$python_bin" ]; then
python_bin="python3"
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
echo "::warning::No usable python3 found; skipping setuptools install (node-gyp fallback unavailable)"
exit 0
fi
if "$python_bin" -c 'import setuptools' >/dev/null 2>&1; then
echo "setuptools already present: $("$python_bin" -c 'import setuptools; print(setuptools.__version__)')"
exit 0
fi
if "$python_bin" - <<'PY'
import sys
raise SystemExit(0 if sys.version_info < (3, 12) else 1)
PY
then
echo "Python <3.12; setuptools distutils shim is not required"
exit 0
fi
if ! "$python_bin" -m pip --version >/dev/null 2>&1; then
if [ "$RUNNER_ENVIRONMENT" = "self-hosted" ]; then
echo "::warning::No usable pip found; skipping setuptools install (node-gyp fallback unavailable)"
exit 0
fi
echo "::error::No usable pip found for setuptools install"
exit 1
fi
"$python_bin" -m pip install setuptools \
|| "$python_bin" -m pip install --user setuptools \
|| "$python_bin" -m pip install --break-system-packages setuptools
- name: Install protoc
# protoc is required even when install-native-deps=false because Rust
# plugins (plugin-{discord,groq,openai,sql,whatsapp}) depend on the
# crates.io `elizaos` crate whose build script invokes prost-build.
# Without protoc, `cargo test` fails before any test runs.
if: ${{ inputs.install-protoc == 'true' && runner.os == 'Linux' }}
shell: bash
env:
FORCE_PROTOC_GITHUB_FALLBACK: ${{ inputs.force-protoc-github-fallback }}
PROTOC_RUNNER_ARCH: ${{ runner.arch }}
run: |
if [ "${FORCE_PROTOC_GITHUB_FALLBACK}" != "true" ] && command -v protoc >/dev/null 2>&1; then
echo "protoc already installed: $(protoc --version)"
exit 0
fi
apt_ok=""
if [ "${FORCE_PROTOC_GITHUB_FALLBACK}" = "true" ]; then
echo "forcing protoc GitHub fallback for setup action self-test"
elif command -v sudo >/dev/null 2>&1 && sudo -n true 2>/dev/null; then
# GitHub-hosted Ubuntu images include Microsoft feeds that are unrelated
# to our protoc install. If those feeds return 403, apt-get update fails
# before it can use the Ubuntu repositories we actually need.
while IFS= read -r source_file; do
if [[ "${source_file}" == *.disabled ]]; then
echo "Microsoft apt source already disabled: ${source_file}"
continue
fi
echo "Disabling unrelated Microsoft apt source: ${source_file}"
sudo mv "${source_file}" "${source_file}.disabled"
done < <(grep -RIl "packages.microsoft.com" /etc/apt/sources.list.d 2>/dev/null || true)
APT_ARGS=(-o Acquire::ForceIPv4=true -o Acquire::Retries=5 -o Acquire::http::Timeout=30)
for attempt in 1 2 3; do
if sudo apt-get "${APT_ARGS[@]}" update && \
sudo apt-get "${APT_ARGS[@]}" install -y --fix-missing protobuf-compiler; then
apt_ok=1
break
fi
sleep $((attempt * 10))
done
fi
if [ -z "$apt_ok" ]; then
# Sudo-free fallback: fetch the pinned protoc release into a user-writable
# dir and put it on PATH. Works on self-hosted runners without passwordless
# sudo or when Ubuntu mirrors fail. The checksum keeps the privileged
# build dependency from trusting an unauthenticated release asset.
echo "apt unavailable/failed; installing protoc from GitHub release (sudo-free)"
protoc_version=27.3
case "${PROTOC_RUNNER_ARCH}" in
X64)
protoc_asset_arch=x86_64
protoc_sha256=6dab2adab83f915126cab53540d48957c40e9e9023969c3e84d44bfb936c7741
;;
ARM64)
protoc_asset_arch=aarch_64
protoc_sha256=bdad36f3ad7472281d90568c4956ea2e203c216e0de005c6bd486f1920f2751c
;;
*)
echo "unsupported Linux runner architecture for protoc fallback: ${PROTOC_RUNNER_ARCH}"
exit 1
;;
esac
dest="${RUNNER_TEMP:-/tmp}/protoc"
rm -rf "$dest"
mkdir -p "$dest"
for attempt in 1 2 3; do
if curl -fsSL --retry 5 --retry-all-errors -o "$dest/protoc.zip" \
"https://github.com/protocolbuffers/protobuf/releases/download/v${protoc_version}/protoc-${protoc_version}-linux-${protoc_asset_arch}.zip"; then
break
fi
if [ "$attempt" -eq 3 ]; then
echo "protoc GitHub release download failed after ${attempt} attempts"
exit 1
fi
sleep $((attempt * 10))
done
echo "${protoc_sha256} $dest/protoc.zip" | sha256sum -c -
unzip -o "$dest/protoc.zip" -d "$dest" 'bin/protoc' 'include/*'
chmod +x "$dest/bin/protoc"
echo "$dest/bin" >> "$GITHUB_PATH"
export PATH="$dest/bin:$PATH"
fi
protoc --version
- name: Install native dependencies
if: ${{ inputs.install-native-deps == 'true' && runner.os == 'Linux' }}
shell: bash
run: |
if ! command -v sudo >/dev/null 2>&1 || ! sudo -n true 2>/dev/null; then
echo "::warning::passwordless sudo unavailable; skipping native apt deps (assuming runner image is pre-provisioned)"
else
# Some CI runners intermittently fail reaching Ubuntu mirrors over IPv6.
# Force IPv4 and retry apt commands so CI setup is resilient. Also
# disable unrelated Microsoft feeds because their 403s make apt-get
# update fail before Ubuntu packages can install.
while IFS= read -r source_file; do
if [[ "${source_file}" == *.disabled ]]; then
echo "Microsoft apt source already disabled: ${source_file}"
continue
fi
echo "Disabling unrelated Microsoft apt source: ${source_file}"
sudo mv "${source_file}" "${source_file}.disabled"
done < <(grep -RIl "packages.microsoft.com" /etc/apt/sources.list.d 2>/dev/null || true)
APT_ARGS=(-o Acquire::ForceIPv4=true -o Acquire::Retries=5 -o Acquire::http::Timeout=30)
for attempt in 1 2 3; do
if sudo apt-get "${APT_ARGS[@]}" update && \
sudo apt-get "${APT_ARGS[@]}" install -y --fix-missing \
build-essential libcairo2-dev libpango1.0-dev libjpeg-dev \
libgif-dev libpixman-1-dev librsvg2-dev; then
break
fi
if [ "$attempt" -eq 3 ]; then
echo "apt install failed after ${attempt} attempts"
exit 1
fi
sleep $((attempt * 10))
done
fi
- name: Setup Bun
uses: oven-sh/setup-bun@v2
with:
bun-version: ${{ inputs.bun-version }}
- name: Cache Bun install
# Cache the Bun install store on every OS, Linux included (#12338). A miss
# just means no speedup and a cache write failure is non-fatal, so this is
# additive: worst case a cold Linux runner behaves exactly as before while
# warm runners skip re-downloading the dependency tarballs. Pinned by SHA
# (no floating @v5) to avoid the Bun/cache-action drift #12338 targets.
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: ~/.bun/install/cache
key: bun-${{ runner.os }}-${{ inputs.bun-version }}-${{ hashFiles('bun.lock') }}
restore-keys: |
bun-${{ runner.os }}-${{ inputs.bun-version }}-
bun-${{ runner.os }}-
- name: Initialize submodules
if: ${{ inputs.init-submodules == 'true' }}
shell: bash
run: git submodule update --init --recursive
- name: Install dependencies
shell: bash
run: |
install_command="${{ inputs.install-command }}"
for attempt in 1 2; do
install_log="$(mktemp)"
if bash -lc "$install_command" 2>&1 | tee "$install_log"; then
rm -f "$install_log"
exit 0
fi
install_status=${PIPESTATUS[0]}
if grep -q "lockfile had changes, but lockfile is frozen" "$install_log" && \
[[ "$install_command" == *"--frozen-lockfile"* ]]; then
fallback_command="${install_command/--frozen-lockfile/--no-frozen-lockfile}"
echo "::warning::Bun reported frozen lockfile drift; falling back to a non-frozen install."
if bash -lc "$fallback_command"; then
# The frozen install above is the canonical lockfile-drift signal.
# When it fails we fall back to a non-frozen install — but Bun
# canary (this repo tracks `bun-version: canary`) reserializes
# bun.lock byte-for-byte between versions even when the resolved
# dependency graph is identical, so a `git diff` here fails on
# serialization noise rather than real drift. Restore the committed
# lockfile after installing; the dedicated lockfile-drift gates
# (cloud-gateway-*, cache-key-stability) still catch genuine drift.
if ! git diff --quiet -- bun.lock; then
echo "::warning::bun.lock changed during install (bun-canary reserialization); restoring the committed lockfile."
git checkout -- bun.lock
fi
rm -f "$install_log"
exit 0
fi
fi
rm -f "$install_log"
if [ "$attempt" -eq 2 ]; then
exit "$install_status"
fi
echo "Dependency install failed; retrying once after cleaning Bun's install cache."
node packages/scripts/rm-path-recursive.mjs ~/.bun/install/cache
sleep 5
done
env:
# When the system-Python guard above found a usable interpreter,
# `pythonLocation` is never set (actions/setup-python@v6 was skipped),
# so prefer the probed binary and only fall back to the downloaded
# toolcache path.
npm_config_python: ${{ steps.system-python.outputs.python-bin || format('{0}/bin/python3', env.pythonLocation) }}
- name: Materialize bun npm package binary
if: ${{ runner.os != 'Windows' }}
# The default install command uses --ignore-scripts so trustedDependencies
# postinstalls don't run silently. The `bun` npm package ships a placeholder
# binary that errors with "Bun's postinstall script was not run" until its
# own install.js downloads the real binary; run it explicitly so any
# subsequent `bun ...` invocation (including the repo postinstall) works.
#
# Bun's hoisted install layout puts the package at
# node_modules/.bun/bun@<ver>/node_modules/bun/install.js rather
# than the legacy node_modules/bun/install.js, so check the
# hoisted path too. Recent bun versions nest deeper (the cache
# uses content-addressable hash dirs); raised maxdepth from 4 to
# 8 and added a final unbounded fallback so every layout we've
# seen in the wild gets caught.
shell: bash
run: |
install_js=""
if [ -f node_modules/bun/install.js ]; then
install_js="node_modules/bun/install.js"
elif [ -d node_modules/.bun ]; then
install_js="$(find node_modules/.bun -maxdepth 8 -path '*/bun@*/node_modules/bun/install.js' -print -quit 2>/dev/null)"
fi
# Final fallback: scan the whole node_modules tree (slower but
# exhaustive). Skip if we already found one above.
if { [ -z "$install_js" ] || [ ! -f "$install_js" ]; } && [ -d node_modules ]; then
install_js="$(find node_modules -path '*/bun/install.js' -print -quit 2>/dev/null)"
fi
if [ -n "$install_js" ] && [ -f "$install_js" ]; then
echo "Running bun postinstall: $install_js"
(cd "$(dirname "$install_js")" && node install.js)
else
echo "::warning::No bun/install.js found; downstream bun run calls may fail"
fi
- name: Run repository postinstall
if: ${{ inputs.run-postinstall == 'true' }}
shell: bash
run: bun run postinstall
env:
SKIP_AVATAR_CLONE: ${{ inputs.skip-avatar-clone == 'true' && '1' || '' }}
ELIZA_NO_VISION_DEPS: ${{ inputs.no-vision-deps == 'true' && '1' || '' }}
- name: Ensure workspace + native-plugin symlinks (postinstall safety net)
# `bun run postinstall` chains `bun scripts/patch-*` steps before
# `ensure-workspace-symlinks.mjs` / `ensure-native-plugins-linked.mjs`;
# if a `bun` invocation fails partway (e.g. the placeholder binary),
# those symlinks never get created and downstream vitest jobs hit
# "Failed to resolve entry for @elizaos/<workspace-pkg>". Run them
# explicitly with `node` (always available) so the symlinks exist
# regardless of how postinstall fared. Both scripts are idempotent.
shell: bash
run: |
node packages/scripts/ensure-workspace-symlinks.mjs || echo "::warning::ensure-workspace-symlinks failed"
node packages/scripts/ensure-native-plugins-linked.mjs || echo "::warning::ensure-native-plugins-linked failed"
- name: Ensure generated shared i18n data
shell: bash
run: node packages/app-core/scripts/ensure-shared-i18n-data.mjs
@@ -0,0 +1,49 @@
name: "GitHub-native Turbo cache"
description: >-
Restore and save the local Turbo cache (.turbo) through actions/cache, keyed
on the deterministic turbo-cache-key hash. This is the GitHub-native
replacement for the Vercel remote cache — no third-party SaaS, no secrets
required (#12341).
# Adopting jobs run turbo with the local filesystem cache only. A job that uses
# this action must NOT also wire the Vercel remote-cache env — the
# ci-turbo-cache-contract guard forbids mixing the two regimes so cache
# provenance stays unambiguous.
inputs:
cache-dir:
description: "Turbo local cache directory (matches turbo --cache-dir)."
required: false
default: ".turbo"
key-prefix:
description: "Extra namespace for the cache key (e.g. a lane name)."
required: false
default: "turbo"
outputs:
cache-hit:
description: "Whether an exact cache entry was restored."
value: ${{ steps.cache.outputs.cache-hit }}
cache-key:
description: "The deterministic turbo cache key used for this run."
value: ${{ steps.key.outputs.turbo_cache_key }}
runs:
using: "composite"
steps:
- name: Compute deterministic turbo cache key
id: key
shell: bash
run: node packages/scripts/turbo-cache-key.mjs --github-output
- name: Restore and save local turbo cache
id: cache
# Pinned to the same actions/cache SHA the rest of the repo uses.
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: ${{ inputs.cache-dir }}
key: ${{ inputs.key-prefix }}-${{ runner.os }}-${{ steps.key.outputs.turbo_cache_key }}
# Fall back to the newest same-OS entry so a partial hit still warms the
# local cache; turbo re-validates task hashes regardless of restore key.
restore-keys: |
${{ inputs.key-prefix }}-${{ runner.os }}-
+169
View File
@@ -0,0 +1,169 @@
# Certification trust anchor
`certification-public-key.pem` is the Ed25519 public key that the
develop→main promotion gate trusts. A `certification.json` (produced by
`bun run --cwd packages/evidence certify:sign`) is a signed claim that a
holder of the matching private key reviewed a specific evidence bundle for a
specific commit. Verification logic lives in
`packages/evidence/src/certify/sign.ts`; the CI workflow must stay a thin
caller of `certify:verify` and perform no verification logic of its own
(#14546 / #14547).
**Trusted key fingerprint: `3ac9e3e625a9ed2f`** (first 16 hex of sha256 over
the SPKI DER). `packages/evidence/src/certify/committed-key.test.ts` asserts
the committed PEM matches this fingerprint, so an accidental or malicious pem
swap fails the test suite as well as review. This key was provisioned during
the epic #14541 bootstrap; its private half lives in the
`ELIZA_CERT_SIGNING_KEY` repo secret. Rotation (procedure below) is cheap and
may be performed at any time by the repo owner — merging a PR that replaces
the pem and this fingerprint is the act of trusting a new key. There is
deliberately no default trust anchor in code: `certify:verify` requires an
explicit `--pubkey`.
## Trust model — binding rules for the CI gate (#14547)
- **Read the public key from the BASE branch (`main`), never from the PR
head.** The gate must check out `main`'s copy of this file (e.g.
`git show origin/main:.github/certification/certification-public-key.pem`)
and pass it via `--pubkey`. If the gate read the key from the PR head, an
attacker could swap the key and a matching self-signed certification in a
single PR and the gate would happily verify their own signature. Key
changes only become trusted after they land on `main` through review.
- **The private key is never in the repo.** It lives only in the
`ELIZA_CERT_SIGNING_KEY` secret (PEM or base64-wrapped PEM) on the signing
runner, and in the local env/keychain of authorized certifiers. No tool in
this repo writes a private key to disk; `certify:keygen` prints it only
under the explicit `--print-private-key` flag.
- **What a valid signature proves:** a keyholder signed exactly these
verdicts over exactly this bundle manifest (`bundleSha`) for exactly this
commit. **What it does not prove:** that the review was diligent. That is
why the reviewer identity (`{kind, id, model?}`) is part of the signed
payload and must be surfaced in the gate's check summary.
## Verify contract (what the workflow calls)
```bash
bun run --cwd packages/evidence certify:verify -- \
--cert <path/to/certification.json> \
--bundle <bundle-dir> \
--pubkey <base-branch-checkout>/.github/certification/certification-public-key.pem \
--expected-commit <cert.commit> \
--max-age-hours 72 \
--required-tier full \
[--requirements <requirements.json>] \
--json
```
Exit 0 iff valid. `--json` prints the full report on stdout, including
`failures: [{code, message, context}]` with distinct codes
(`schema-invalid | unsigned | bad-signature | wrong-key | stale |
commit-mismatch | bundle-tampered | verdict-failures | verdict-incomplete |
tier-insufficient`); all detectable failures are reported together, not
first-failure-only.
For the required develop→main gate, `--bundle` is mandatory. Without the
bundle, verification can only prove that the JSON was signed by the trusted
key; it cannot prove the signed `bundleSha` matches real artifacts, that every
mechanically-derived lane/analysis subject was reviewed, or that verdict
evidence paths actually exist. Omitting `--bundle` is only for detached
signature inspection, never for promotion. With a bundle, verification also
re-runs the mechanical rollup: mechanically non-pass subjects may be signed as
`fail` or `waived` with notes, but never omitted or signed as `pass`.
## Key rotation
1. An authorized engineer runs
`bun run --cwd packages/evidence certify:keygen -- --print-private-key`
on a trusted machine (never in CI logs).
2. Open a PR replacing `certification-public-key.pem` and the fingerprint in
this README. The PR must be reviewed and merged to `main` like any other
change — merging it is the act of trusting the new key.
3. Update the `ELIZA_CERT_SIGNING_KEY` repo secret (and any local certifier
copies) to the new private key.
4. Certifications signed by the old key stop verifying the moment the new
public key is on `main` (`wrong-key`). Re-certify in-flight promotions.
Compromise response is the same procedure, performed immediately; the old
key needs no explicit revocation because the gate trusts exactly one key —
the one on `main`.
## The CI gate (#14547): `certification-verify.yml`
`.github/workflows/certification-verify.yml` runs on every non-draft PR
targeting `main` and is the promotion gate. What it does, in order:
1. Checks out the PR head commit (not the synthetic merge ref) and reads the
trusted public key from the **base branch** per the trust model above.
2. Locates the certification and bundle: **`certification.json` committed at
the repo root of the promotion branch** and the signed evidence bundle
committed at **`evidence/bundle/`**. Missing either file = red, with
remediation instructions in the check summary. The gate always passes
`--bundle evidence/bundle` so artifact integrity, rollup completeness, and
signed evidence-path membership are re-derived from the committed bundle.
3. Runs `scripts/certification/check-commit-drift.mjs`: the certified commit
must equal the PR head, or be an ancestor whose diff to head touches only
the docs/template allowlist — `docs/**`, `packages/docs/**`, any `*.md`,
plus the certification artifacts themselves (`certification.json`,
`evidence/bundle/**` — those necessarily land after the signed commit and
are protected by the signature and `bundleSha`, not by the drift rule). Any
other source, workflow, policy, or config drift = re-certify.
4. Runs `certify:verify` (the contract above) with
`--expected-commit <cert.commit> --max-age-hours 72 --required-tier full`.
5. Writes a check summary with the reviewer identity, tier, age, signed
verdict table (waived rows surfaced with notes), drifted paths, trusted
key fingerprint, and every failure code verbatim. Verification only — the
gate never performs evidence *review*.
### Branch protection (admins)
Add the required status check **`verify-certification`** (the job name in
`certification-verify.yml`) to `main`'s branch protection. This is a manual
admin step — the workflow does not (and must not) edit branch protection.
Keep the name stable; renaming the job orphans the required check and blocks
every promotion. The companion `certification-verify-selftest` job is CI for
the gate's own plumbing (runs only on PRs touching it) and must **not** be
made a required check.
**Bootstrap:** until this directory's public key and `packages/evidence`
exist on `main`, the gate fails closed on every promotion PR ("no trusted
public key on main"). The first promotion that carries the certification
stack to `main` needs a one-time admin bypass; every promotion after it is
fully gated.
**Residual risk (accepted):** `pull_request`-triggered workflows execute the
PR's own merged workflow definition, so a promotion PR that edits the gate
files could neuter the check. The drift check refuses to allowlist those
paths, the diff is plainly visible to the human merger, and promotion PRs
are human-reviewed by definition. Hardening to `pull_request_target` (base
branch runs the workflow) is a possible follow-up.
### Remediation runbook (gate is red)
1. Preferred: dispatch the vast certification workflow for the promoted
commit; it produces the bundle, review, and signed `certification.json`.
2. Local fallback (requires a certifier holding `ELIZA_CERT_SIGNING_KEY`):
```bash
bun run --cwd packages/evidence bundle:create -- --tier full
bun run --cwd packages/evidence certify:rollup -- --bundle <bundle-dir> --out verdicts.json
# review verdicts.json by hand — waivers require notes
bun run --cwd packages/evidence certify:sign -- --bundle <bundle-dir> \
--verdicts verdicts.json --reviewer-id <you> --reviewer-kind human
cp <bundle-dir>/certification.json ./certification.json # commit on the promotion branch
rm -rf evidence/bundle && mkdir -p evidence && cp -R <bundle-dir> evidence/bundle
```
3. `stale` (>72h) or `commit-mismatch`/drift failures always mean the same
thing: re-run certification at the current head. There is no override
flag by design.
## Break-glass (emergencies)
Push to `main` is a production deploy, so the gate must be watertight — but
it must not brick emergencies. The break-glass path is **not a code path**:
a repository admin bypasses the required `certification-verify` check via
branch-protection admin override. Every such bypass is visible in the
GitHub audit log; there is deliberately no in-repo flag, env var, or
alternate verification mode that skips signature checks. The required check
being bypassed is **`verify-certification`** (see the CI gate section
above) — that is the exact string admins bypass and auditors grep for.
@@ -0,0 +1,3 @@
-----BEGIN PUBLIC KEY-----
MCowBQYDK2VwAyEAdfTNwBkS2OyFmk2CBxypZoc5u7p0l/5QRWcbJsiAtSs=
-----END PUBLIC KEY-----
+4
View File
@@ -0,0 +1,4 @@
{
"$comment": "Single source of truth for the pinned Bun version on every frozen-lockfile CI install path. Floating `canary`/`latest` reserializes bun.lock to lockfileVersion 2 and breaks `bun install --frozen-lockfile` (#11184/#9454), and the repo's packageManager `bun@1.4.0-canary.1` is unresolvable in CI (GH+npm 404) — so the required test/build gates pin a concrete version here instead. Enforced by packages/scripts/ci-bun-version-contract.mjs: every concrete `bun-version`/`BUN_VERSION` pin in .github/workflows must equal `version`, and the gate workflows in that script's GATE_WORKFLOWS list must stay pinned to it and never float back to canary/latest. To bump: change `version` here and update the gate workflows in the SAME PR — the contract fails the branch if they drift apart.",
"version": "1.3.14"
}
+27
View File
@@ -0,0 +1,27 @@
{
"$comment": "Coverage floor for the Windows CI lane (#13402). Every command here must stay wired in some jobs.windows.strategy.matrix.include[].commands list in .github/workflows/windows-ci.yml. The Windows lane is a deliberately narrow subset of the Linux gates (see WINDOWS.md), so these suites are the only Windows proof for the runtime/dashboard/shared TypeScript packages — dropping one silently regresses coverage with a still-green pipeline. Enforced by packages/scripts/ci-windows-command-coverage-contract.mjs: adding a command is always allowed (this is a floor, not an exact match); to intentionally retire one, delete it from the matrix AND from this list in the SAME PR so the reduction is reviewable in the diff.",
"commands": [
"node packages/scripts/run-turbo.mjs run typecheck --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/cloud-shared --concurrency=4",
"bun run --cwd packages/core test",
"bun run --cwd packages/shared test",
"bun run --cwd packages/app-core test",
"bun run --cwd packages/elizaos test",
"bun run --cwd packages/cloud/shared test",
"bun run --cwd packages/scenario-runner test",
"bun run --cwd packages/vault test",
"bun run --cwd packages/security test",
"bun run --cwd plugins/plugin-coding-tools test",
"bun run --cwd plugins/plugin-elizacloud test",
"bun run --cwd plugins/plugin-discord test",
"bun run --cwd plugins/plugin-anthropic test",
"bun run --cwd plugins/plugin-openai test",
"bun run --cwd plugins/plugin-app-control test",
"bun run --cwd plugins/plugin-task-coordinator test",
"bun run --cwd plugins/plugin-browser test",
"node packages/scripts/run-turbo.mjs run build --filter=@elizaos/core --filter=@elizaos/shared --filter=@elizaos/agent --concurrency=4",
"node packages/scripts/run-bash-linux-only.mjs scripts/verify-riscv64-buildpaths.sh",
"node packages/scripts/run-python.mjs --version",
"node packages/scripts/test-cloud-run.mjs",
"node packages/scripts/clean-stray-dts.mjs"
]
}
+85
View File
@@ -0,0 +1,85 @@
name: elizaOS CodeQL
# Keep the default CodeQL query selection. This config narrows what the JS/TS
# extractor ingests so the build-mode:none database stops OOMing (exit 137 in
# the dataflow queries). For interpreted/no-build languages, paths-ignore
# filters EXTRACTED files (not just reported results), so excluding generated
# build output and non-product trees genuinely shrinks the DB + mmap footprint.
# Every glob below targets build artifacts, vendored-dependency trees, or
# non-product (research/eval/demo) code. ZERO product server/runtime/security
# source is excluded: packages/core, agent/src, app-core/src, app/src, ui/src,
# cloud/**, security, vault, feed/packages/api, and plugins/** all stay scanned.
paths-ignore:
# WHY: this benchmark fixture is deliberately invalid TypeScript; the Solana
# runner test writes bad code here to assert syntax failures are reported
# cleanly. Analyzing it adds noisy parser diagnostics without adding security
# coverage.
- packages/benchmarks/solana/solana-gym-env/voyager/skill_runner/_test_bad.ts
# WHY: installed dependencies, not first-party source (CodeQL skips these by
# default; listed explicitly to also drop the vendored llama.cpp webui
# node_modules JS the extractor would otherwise ingest).
- '**/node_modules/**'
# WHY: compiled JS build output mirroring src/*.ts (the real source stays).
# #1 OOM driver by file count (packages/app-core/scripts/bun-riscv64/dist
# alone is ~88k generated .js).
- '**/dist/**'
# WHY: compiled bundles, not source (e.g. packages/app/electrobun/build, ~18.5k
# generated .js). The app's real UI source in packages/app/src stays.
- '**/build/**'
# WHY: Next.js build output; each SSR route bundle inlines ~22MB of deps
# (packages/feed/**/.next). The feed's true source in src + api stays.
- '**/.next/**'
# WHY: repeated example-app build output (numbered .next-build-NNNN dirs).
- '**/.next-build-*/**'
# WHY: Turbo task cache, never source.
- '**/.turbo/**'
# WHY: test-coverage reports, generated artifacts.
- '**/coverage/**'
# WHY: generic build/export output dir, generated not authored.
- '**/out/**'
# WHY: Python virtualenvs ship vendored JS (trame, litellm swagger-ui) that
# bloats ingest; never product source.
- '**/.venv/**'
- '**/venv/**'
# WHY: vendored Python deps (site-packages) that include bundled JS; not source.
- '**/site-packages/**'
# WHY: Storybook build output (packages/ui/storybook-static); ui's real source
# in packages/ui/src stays.
- '**/storybook-static/**'
# WHY: fully bundled mobile agent build output (packages/agent/dist-mobile*),
# 107MB across 2 files; the real agent source in packages/agent/src stays.
- '**/dist-mobile/**'
- '**/dist-mobile-ios/**'
# WHY: Playwright HTML run reports, generated test artifacts.
- '**/playwright-report/**'
# WHY: the web bundle baked into the iOS IPA at build time — a compiled copy
# of app source already analyzed in packages/app/src.
- '**/ios/App/App/public/**'
# WHY: the web bundle baked into the Android APK at build time — a compiled
# copy of app source already analyzed elsewhere.
- '**/android/app/src/main/assets/public/**'
- '**/platforms/android/app/src/main/assets/public/**'
# WHY: transient desktop-shell build-probe bundles under platforms/*/tmp
# (~21MB each); pure throwaway build output.
- '**/platforms/*/tmp/**'
# WHY: minified bundles are unreadable generated output, not source.
- '**/*.min.js'
# WHY: vendored Bun-from-source toolchain build cache (~88k generated .js); the
# single largest OOM contributor. Lives under scripts/, not src/.
- packages/app-core/scripts/bun-riscv64/**
# WHY: ML research code + datasets + vendored .venv JS; not shipped product and
# not a JS/TS attack surface (Python pipelines).
# WHY: eval/benchmark harnesses, fixtures, and datasets; not deployed product,
# no live untrusted-input surface.
- packages/benchmarks/**
# WHY: 37 standalone reference/demo apps (documentation samples), not the
# shipped product handling untrusted input.
- packages/examples/**
# WHY: device/OS images, OS landing page, robotics; not a server/runtime
# untrusted-input surface.
- packages/os/**
# WHY: Docusaurus docs site; marketing/docs, not product runtime.
- packages/docs/**
# WHY: marketing site; not an untrusted-input runtime surface.
- packages/homepage/**
+48
View File
@@ -0,0 +1,48 @@
version: 2
updates:
# JS/TS workspace (root). Bun monorepo — pulls all workspace package.json files.
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: "weekly"
open-pull-requests-limit: 5
groups:
dev-dependencies:
dependency-type: "development"
ignore:
# isomorphic-git@1.x requires ignore@^5; the `ignore` dep in @elizaos/agent
# exists only to satisfy it. v6/v7 break isomorphic-git's GitIgnoreManager
# (git.add fails — see closed PR #8483). Cap at major 5 until
# isomorphic-git supports ignore@7.
- dependency-name: "ignore"
versions: [">= 6.0.0"]
# GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
# Python benchmarks (kept on weekly cadence; demo / research code)
- package-ecosystem: "pip"
directory: "/packages/benchmarks/OSWorld"
schedule:
interval: "weekly"
open-pull-requests-limit: 2
ignore:
# Transitive via wandb; advisory has no patched version yet (opened 2026-04).
- dependency-name: "gitpython"
# Transitive via fabric -> paramiko; libsodium 1.0.20 bundled in PyNaCl 1.6.x
# already includes the disallowed-inputs fix.
- dependency-name: "pynacl"
- package-ecosystem: "pip"
directory: "/packages/benchmarks/solana/solana-gym-env"
schedule:
interval: "weekly"
open-pull-requests-limit: 2
# NOTE: cargo (Rust) example entries were removed — commit b69d70632b made the
# examples TypeScript-only and deleted the packages/examples/*/rust/* trees.
# The stale cargo entries made Dependabot abort every run with
# "Cargo.toml not found". Re-add cargo blocks only if a real Cargo.toml returns.
+73
View File
@@ -0,0 +1,73 @@
# Auto-labels PRs based on changed file paths
# Used by actions/labeler in auto-label.yml
core:
- changed-files:
- any-glob-to-any-file:
- 'packages/agent/**'
- 'packages/app-core/src/runtime/**'
- 'packages/app-core/src/cli/**'
- 'packages/app-core/src/config/**'
connector:
- changed-files:
- any-glob-to-any-file:
- 'plugins/plugin-telegram/**'
- 'plugins/plugin-discord/**'
- 'plugins/plugin-slack/**'
- 'plugins/plugin-imessage/**'
- 'plugins/plugin-whatsapp/**'
ui:
- changed-files:
- any-glob-to-any-file:
- 'packages/app/**'
- 'packages/ui/**'
- 'packages/homepage/**'
- '**/*.css'
- '**/*.scss'
tests:
- changed-files:
- any-glob-to-any-file:
- '**/*.test.ts'
- 'packages/app-core/test/**'
- 'test/**'
- 'vitest*.config.*'
ci:
- changed-files:
- any-glob-to-any-file:
- '.github/**'
docs:
- changed-files:
- any-glob-to-any-file:
- '**/*.md'
- 'packages/docs/**'
build:
- changed-files:
- any-glob-to-any-file:
- 'package.json'
- 'tsconfig*.json'
- 'tsdown.config.*'
- 'packages/app-core/scripts/**'
plugins:
- changed-files:
- any-glob-to-any-file:
- 'plugins/**'
- 'packages/native/plugins/**'
electron:
- changed-files:
- any-glob-to-any-file:
- 'packages/app/electrobun/**'
deploy:
- changed-files:
- any-glob-to-any-file:
- 'deploy/**'
- 'Dockerfile*'
- 'docker-compose*'
+186
View File
@@ -0,0 +1,186 @@
<!-- Fill every visible section. Keep every evidence row visible; use N/A with a concrete reason when a row does not apply. -->
# Relates to
<!--
Link the issue, ticket, or Project card. For agent/kanban work, follow
CONTRIBUTING.md and keep the Project item status current.
-->
Definition of Done: full standard in [`CONTRIBUTING.md`](../CONTRIBUTING.md).
- [ ] This PR targets `develop` and is rebased onto the latest `origin/develop`
with zero conflicts (`git fetch origin && git rebase origin/develop`).
- [ ] `bun install` and `bun run verify` were run after sync, or any failure is
recorded below with the exact unrelated blocker.
- [ ] A reviewer can confirm the change works without reading the code, from the
evidence attached below.
# Sync with develop
- [ ] Rebased/merged onto the latest `origin/develop`; zero conflicts.
- [ ] `bun run verify` passes post-sync, or the exact unrelated blocker is
documented in **Known gaps / failures** below.
<!-- This risks section must be filled out before the final review and merge. -->
# Risks
<!--
Low, medium, large. List what kind of risks and what could be affected.
-->
# Background
## What does this PR do?
## What kind of change is this?
<!--
Bug fixes (non-breaking change which fixes an issue)
Improvements (misc. changes to existing features)
Features (non-breaking change which adds functionality)
Updates (new versions of included code)
-->
<!-- This "Why" section is most relevant if there are no linked issues explaining why. If there is a related issue, it might make sense to skip this why section. -->
<!--
## Why are we doing this? Any context or related work?
-->
# Documentation changes needed?
<!--
My changes do not require a change to the project documentation.
My changes require a change to the project documentation.
If documentation change is needed: I have updated the documentation accordingly.
-->
<!-- Please show how you tested the PR. This will really help if the PR needs to be retested and probably help the PR get merged quicker. -->
# Testing
## Where should a reviewer start?
## Detailed testing steps
<!--
None: Automated tests are acceptable.
-->
<!--
- As [anon/admin], go to [link]
  - [do action]
  - verify [result]
-->
# Evidence Gate
Any change testable on the frontend is not mergeable without a video walkthrough,
before/after screenshots, and logs. If you did not attach them, say why.
Attach each applicable artifact **inline in this PR** (drag-and-drop into the
description or a comment), or write `N/A - <reason>` on the row. Do not leave
evidence rows blank. Videos must be **MP4** (GitHub renders them inline);
prefer **JPG over PNG** for screenshots. Do not commit evidence files to the
repo.
<!-- evidence-row:before-screenshots -->
- [ ] Before full-page screenshots are attached for every affected UI surface
(desktop and mobile), or marked `N/A - <reason>`.
<!-- evidence-row:after-screenshots -->
- [ ] After full-page screenshots are attached for every affected UI surface
(desktop and mobile), or marked `N/A - <reason>`.
<!-- evidence-row:walkthrough-video -->
- [ ] A video walkthrough of the complete user flow is attached, or marked
`N/A - <reason>`.
<!-- evidence-row:backend-logs -->
- [ ] Backend logs show the real code path firing end to end, or are marked
`N/A - <reason>`.
<!-- evidence-row:frontend-logs -->
- [ ] Frontend console and network logs show the request/response and state
change, or are marked `N/A - <reason>`.
<!-- evidence-row:llm-trajectory -->
- [ ] Real-LLM trajectory is attached for agent/action/provider/prompt/model
changes, or marked `N/A - <reason>`.
<!-- evidence-row:domain-artifacts -->
- [ ] Domain artifacts are attached where applicable (DB rows, memories,
scheduled tasks, wallet/on-chain output, generated files, audio, etc.), or
marked `N/A - <reason>`.
# Evidence Details
## Real LLM-call trajectory
For agent/action/provider/prompt/model changes, use a real live-model run, not
the deterministic proxy. Produce with:
```bash
packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out.json>
```
Link the JSON report, run viewer, native jsonl, or write `N/A - <reason>`.
## Backend + frontend logs
Backend: structured logger lines ([ClassName] …) showing the code path firing end to end.
Frontend: console + network trace showing the request/response and state change.
Paste here inline (wrap long output in a `<details>` block), or write
`N/A - <reason>`.
## Screenshots (before / after) + video walkthrough
Full-page before AND after screenshots are required for any UI change. Include a
video click-through of the flow.
```bash
bun run evidence:doctor (check capture tools; prints fixes for any missing)
bun run test:e2e:record (general E2E recordings)
bun run --cwd packages/app audit:app (app + cloud UI — REQUIRED for UI changes)
```
A UI-touching diff (rendered `.tsx`/`.css`/`.svg` under `packages/app`,
`packages/ui`, `apps/app`, …) MUST attach real screenshot/video/OCR artifacts —
the CI evidence gate rejects `N/A` on those rows for such diffs, label or not.
If a capture tool is missing, `evidence:doctor` prints the install command;
install it rather than marking evidence `N/A`.
### Before
### After
### Walkthrough video
Or write `N/A - <reason>`.
## Audio / voice walkthrough
For voice / transcript / TTS / STT / omnivoice changes, attach captured audio of
the real round-trip plus a narrated walkthrough. Or write `N/A - <reason>`.
## Known gaps / failures
List any command failure, missing artifact, unavailable device, unavailable live
service, or evidence row marked N/A. Include the exact reason and why it is not a
blocker for this PR.
<!-- If there is anything about the deployment, please make a note. -->
<!--
# Deploy Notes
-->
<!--  Copy and paste command line output. -->
<!--
## Database changes
-->
<!--  Please specify deploy instructions if there is something more than the automated steps. -->
<!--
## Deployment instructions
-->
<!-- If you are on Discord, please join https://discord.gg/ai16z and state your Discord username here for the contributor role and join us in #development-feed -->
<!--
## Discord username
-->
+51
View File
@@ -0,0 +1,51 @@
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"description": "Shared Renovate configuration preset for elizaOS plugins",
"extends": ["config:recommended"],
"packageRules": [
{
"description": "Pin @elizaos/* packages and check daily for new releases",
"groupName": "elizaOS Core Dependencies",
"matchPackagePatterns": ["^@elizaos/"],
"rangeStrategy": "pin",
"schedule": ["every day"],
"separateMajorMinor": true,
"automerge": false
},
{
"description": "Group TypeScript ecosystem updates",
"groupName": "TypeScript",
"matchPackagePatterns": ["^typescript$"]
},
{
"description": "Group Biome linting updates",
"groupName": "Linting",
"matchPackagePatterns": ["^@biomejs/"]
},
{
"description": "Group build tools",
"groupName": "Build Tools",
"matchPackageNames": ["vite", "rollup"],
"matchPackagePatterns": [
"^@vitejs/",
"^@rollup/",
"^vite-plugin-",
"^rollup-plugin-"
]
}
],
"timezone": "UTC",
"schedule": ["every day"],
"prHourlyLimit": 5,
"prConcurrentLimit": 15,
"rangeStrategy": "pin",
"separateMajorMinor": true,
"separateMinorPatch": false,
"dependencyDashboard": true,
"dependencyDashboardTitle": "🤖 Renovate Dependency Dashboard",
"commitMessagePrefix": "chore(deps):",
"semanticCommits": "enabled",
"automerge": false,
"platformAutomerge": false,
"rebaseWhen": "conflicted"
}
+16
View File
@@ -0,0 +1,16 @@
#!/usr/bin/env bash
set -euo pipefail
if [ "$#" -eq 0 ]; then
echo "usage: $0 <browser> [browser...]" >&2
exit 2
fi
if [ "${RUNNER_OS:-}" = "Linux" ] || [ "$(uname -s 2>/dev/null || true)" = "Linux" ]; then
if command -v sudo >/dev/null 2>&1 && sudo -n true 2>/dev/null; then
exec bunx playwright install --with-deps "$@"
fi
echo "::notice::passwordless sudo unavailable; installing Playwright browsers without OS deps"
fi
exec bunx playwright install "$@"
+159
View File
@@ -0,0 +1,159 @@
# CI Workflow Changelog
This changelog tracks meaningful CI policy and workflow-architecture changes.
It is intentionally scoped to `.github/workflows` so product/package changelogs
do not have to carry CI-only history.
## 2026-07-04
### Changed
- Migrated the whole repo off the Vercel SaaS Turbo remote cache onto the
GitHub-native cache (#12341, epic #12191 phase 4). Removed every
`TURBO_TOKEN` / `TURBO_TEAM` / `TURBO_CACHE: remote:rw` env from all
workflows (108 lines across 11 files) and routed `setup-bun-workspace`'s
Turbo cache through the pinned `turbo-cache-github` shim, whose single
per-hash key replaces the former per-job-name key that fragmented the cache
across ~30 job names.
Why: project guidance is GitHub-native caching only; the SaaS env was
redundant with the `.turbo` `actions/cache` layer that already ran in the
shared setup action. `ci-workflow-dedup-contract.mjs` now **forbids** the
SaaS env anywhere and **requires** the GitHub-native cache on the publish
paths (nightly/release) — the inverse of its previous assertion, which
pinned the SaaS wiring.
- PR-lane `typecheck` (`ci.yaml`) and `lint` (`quality.yml`) now run through
`turbo --affected` scoped to the PR merge base (`TURBO_SCM_BASE`), with full
history fetched so scoping resolves; `push`/`merge_group` still run the full
graph. A shallow clone degrades to running everything, so scoping never
under-checks.
### Added
- `develop-exhaustive.yml` (#12342, epic #12191 phase 5): an un-cancellable
scheduled orchestrator (06:00/18:00 UTC, dedicated concurrency group) that
invokes every platform lane test.yml's schedule does not cover — Windows,
mobile, scenario, the three UI gates, keyless harness, docker, dev
onboarding, electrobun/desktop — via `workflow_call`, then runs the matrix
proof. A skipped or failed reusable lane fails the run (a coverage gap is not
a pass). Added a bare `workflow_call:` trigger to each of those 10 workflows
so they are reusable.
Why: the scheduled exhaustive lane is the "prove develop runs the full
matrix" DoD. `ci-full-matrix-proof.mjs` + `ci-lane-manifest.json` gained a
`reusableWorkflows` check so dropping a lane's `uses:` or a workflow's
`workflow_call` trigger fails the proof statically, before the run.
## 2026-06-29
### Changed
- Split automatic branch ownership between `ci.yaml` and `test.yml`:
`ci.yaml` remains the main-branch gate, while `test.yml` now runs
automatically on develop plus manual/scheduled invocations.
Why: both workflows were auto-running overlapping build/test work on `main`.
The zero-key PR gate for main remains covered by `scenario-pr.yml`, and
develop keeps the broader `Tests` orchestrator.
- Switched the root `test:plugins` script from a direct Turbo sweep to the
shard-aware cross-package test runner at concurrency 3.
Why: `TEST_SHARD` is implemented in `run-all-tests.mjs`. Routing the plugin
lane through that runner makes the matrix real instead of cosmetic, while
preserving the existing bounded concurrency policy.
### Added
- Added `packages/scripts/ci-workflow-dedup-contract.mjs`.
Why: this locks the branch-trigger split and verifies nightly/release keep the
Turbo remote-cache environment required by #10096.
- Added develop/main quality gates for the prompt-secret scan, UI determinism
audit, and build-enabled lint.
Why: `ci.yaml` still runs only for `main`, while the default PR target is
`develop`. These checks now run through `quality.yml`, which already covers
develop PRs.
- Split the plugin test lane into a four-way `TEST_SHARD` matrix with a stable
aggregate `Plugin Tests` check.
Why: the cross-package runner already had deterministic shard membership, but
no workflow used it. The plugin lane is the long pole in `test.yml`; sharding
it cuts wall-clock without reducing package coverage.
## 2026-06-14
### Added
- Added `packages/scripts/ci-path-gate.mjs` as the shared PR path classifier for
expensive workflows.
Why: repeated inline shell classifiers made it too easy for workflows to drift
apart, and reviewers could not see a consistent explanation for why a lane ran
or skipped.
- Added path-gate summaries to show changed files, selected lanes, and the path
or label reason for each lane.
Why: fast CI is only useful if contributors and maintainers trust the skip
decision. The summary turns the decision into reviewable evidence.
- Added force labels including `ci:full`, `ci:e2e`, `ci:zero-key`,
`ci:server`, `ci:client`, `ci:plugins`, `ci:cloud`, `ci:docker`,
`ci:mobile`, `ci:ios`, `ci:android`, `ci:desktop`, `ci:windows`, and
`ci:dev-smoke`.
Why: maintainers need a no-code way to request broader coverage when a change
is risky, cross-cutting, or ambiguous.
- Added `packages/scripts/ci-path-gate.self-test.mjs` and run it before the
`Tests` workflow consumes classifier outputs.
Why: the classifier is now part of the quality gate. Testing it in CI prevents
a future edit from silently skipping coverage that should have run.
### Changed
- Replaced inline path filters in `test.yml` and `scenario-pr.yml` with the
shared classifier.
Why: one implementation is easier to audit, document, and extend than several
workflow-local shell snippets.
- Added classifier jobs to Docker, mobile, dev smoke, Windows dev smoke, and
Windows desktop preload smoke workflows.
Why: these lanes are valuable but expensive. Running them only for relevant
PRs keeps feedback fast while still preserving push/manual coverage.
- Split deterministic zero-key E2E work into named parallel slices while keeping
the visible `Zero-Key Deterministic E2E` aggregate check.
Why: a single serial E2E log made failures slow to reach and hard to triage.
Parallel slices shorten wall-clock time and make the failing surface obvious
without removing the aggregate gate reviewers already understand.
- Moved `coverage-gate` dependency setup behind changed-test detection and
switched it to the shared Bun workspace setup.
Why: the coverage gate is advisory for changed Bun-native tests. Docs-only and
no-test PRs should not install the whole workspace or fail on unrelated
registry/install noise, while test-bearing PRs should use the same
lockfile-validating setup path as the rest of CI.
### Preserved
- Push, scheduled, and manual runs keep broad/default behavior.
Why: those runs protect branch health, release readiness, and periodic
confidence. PR path gates optimize contributor feedback, not the repository's
deeper safety net.
- The split E2E jobs keep the previous substantive commands.
Why: this change is a CI ergonomics and parallelism improvement, not a
reduction in coverage.
+289
View File
@@ -0,0 +1,289 @@
# CI/CD Workflows
This directory contains GitHub Actions workflows for the elizaOS project (v2.0.0).
## Workflow Overview
| Workflow | Trigger | Purpose |
|----------|---------|---------|
| `ci.yaml` | Push/PR to main | Main-specific CI - typecheck, tests, lint, build, dev startup |
| `test.yml` | Push/PR to develop, manual, schedule | Broader develop tests plus required zero-key deterministic E2E; live jobs are separate |
| `quality.yml` | Push/PR to main/develop, manual | Develop/main quality gates: format, type-safety ratchet, prompt-secret scan, UI determinism, lint |
| `scenario-pr.yml` | PR to main/develop, manual | Secret-free deterministic scenario/browser E2E gate |
| `scenario-matrix.yml` | Develop/manual opt-in | Real-service scenario matrix; not a PR gate |
| `pr.yaml` | PR opened/edited | PR title validation |
| `release.yaml` | Push to main, Release | NPM beta/production package releases |
| `claude.yml` | @claude mentions | Interactive Claude assistance |
| `claude-code-review.yml` | PR opened | Automated code review |
| `claude-security-review.yml` | PR opened | Security-focused review |
| `codeql.yml` | Push/PR to main, Weekly | Static security analysis |
| `docs-ci.yml` | PR (docs paths), Manual | Documentation quality checks |
| `build-agent-image.yml` | Push develop/main, Release, Manual | Docker image builds (`:develop`, `:stable`, `:latest`, release tags) |
| `tee-build-deploy.yml` | Push to main, Manual | TEE deployment to Phala Cloud |
| `weekly-maintenance.yml` | Weekly, Manual | Dependency/security audits |
| `jsdoc-automation.yml` | Manual | JSDoc generation |
## Release Workflows
### Alpha Tags
Alpha version tags are tags only. They do not publish NPM packages, run packaging
CI, or create GitHub Release entries.
### NPM Beta/Production Packages (`release.yaml`)
Publishes TypeScript/JavaScript packages to NPM.
**Triggers:**
- Push to `main` → Beta release (`@beta` tag)
- GitHub Release created → Production release (`@latest` tag)
**Packages:** All `@elizaos/*` packages in the monorepo
## Test Workflows
### Linux Runner Policy
The heavy develop **test lanes** in `test.yml` run on the self-hosted
`self-hosted, hetzner-robot` pool (GitHub-hosted minutes are billing-frozen for
this org, #13481). Everything the **merge gate** depends on to *reach a
conclusion* stays GitHub-hosted so a drained fleet can never wedge develop:
- **Path classifiers** (`Classify changed paths`) across `test.yml`,
`scenario-pr.yml`, `dev-smoke.yml`, `docker-ci-smoke.yml`,
`mobile-build-smoke.yml`, `windows-dev-smoke.yml`, and
`windows-desktop-preload-smoke.yml` run on `ubuntu-24.04`. They are git-diff +
node scripts with no self-hosted needs; pinning them to the fleet (#8501) once
left every downstream job queued indefinitely and gridlocked develop.
- **`ci-ok`** (the merge queue's sole required context), its
`plugin-tests-status` roll-up, and the hosted **`merge-quality-gate`** all run
on `ubuntu-24.04`.
Two SPOF guards, enforced by `packages/scripts/ci-merge-gate-contract.mjs` (run
in the `changes` job, #13617):
1. **Fleet-drain toggle.** Every self-hosted lane in `test.yml` reads
`runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}`.
Unset/anything-but-`false` keeps the current self-hosted placement; there is
no way to probe fleet health from a `runs-on:` expression, so during an
outage an admin sets repo **variable** `HETZNER_FLEET_ONLINE=false` once and
the whole workflow falls back to hosted — one flip unblocks the entire queue
instead of per-PR admin-bypass. Keep the runner-agnostic step hardening (no
`sudo`-only install/cleanup) so lanes run on either runner type.
2. **Hosted quality parity.** `merge-quality-gate` runs the same lint /
`format:check` / repo-wide `typecheck` / gitleaks secret scan that guard
`main`, and `ci-ok` needs it — so a lint, type, format, or committed-secret
regression is refused by the merge queue on develop, not just on `main`. It
runs on `merge_group` + develop `push`. The lightweight `develop-pr.yml`
lint job also runs `format:check`, so formatting fails on the PR even when a
busy push wave supersedes post-merge quality runs (#15959).
CodeQL is a separate exception: trusted push, scheduled, and manual CodeQL runs
use `self-hosted, Linux, X64, hetzner-robot` because full JavaScript analysis is
disk-bound and has exhausted GitHub-hosted runners during the `PolynomialReDoS`
dataflow query. Pull-request CodeQL remains GitHub-hosted so forked code never
executes on self-hosted machines. Keep the full CodeQL query surface intact;
move capacity around rather than weakening security coverage. The CodeQL config
may ignore deliberately invalid negative-test fixtures, but not real source
files; those fixtures should stay covered by their owning tests.
GPU / KVM / macOS jobs (labels `gpu-cuda-12.6`, `kvm`, `eliza-e2e-macos`) are a
separate purpose-built fleet and are unaffected by this policy.
### PR Path Gates
PR workflows use `packages/scripts/ci-path-gate.mjs` to keep expensive lanes
targeted. Each classifier job writes a GitHub step summary showing:
- which files changed
- which lanes will run
- which path or label caused each lane to run
Maintainers can force specific lanes with labels:
| Label | Effect |
|-------|--------|
| `ci:full` | Run every path-gated lane in workflows that honor the shared gate |
| `ci:e2e` / `ci:zero-key` | Run deterministic zero-key E2E lanes |
| `ci:scenario` | Run `scenario-pr.yml` deterministic scenario/browser E2E |
| `ci:server` | Run server tests |
| `ci:client` | Run client tests |
| `ci:plugins` | Run plugin tests |
| `ci:cloud` | Run cloud live E2E where secrets are configured |
| `ci:docker` | Run Docker CI smoke |
| `ci:mobile` / `ci:ios` / `ci:android` | Run mobile smoke, or one mobile platform |
| `ci:desktop` / `ci:windows` | Run desktop and Windows smoke lanes |
| `ci:dev-smoke` | Run the `bun run dev` onboarding smoke |
Push, scheduled, and manual runs keep their broader/default behavior; the path
gate mainly keeps PR feedback fast and explainable.
Why this exists:
- OSS contributors should get useful feedback quickly without waiting on
unrelated mobile, Docker, desktop, Windows, or browser-heavy lanes.
- Maintainers should be able to see why a lane ran or skipped from the job
summary, without reverse-engineering shell conditionals.
- The quality gate should stay equivalent for affected code. Path gates decide
which surface is relevant; they do not replace the tests for that surface.
- Push, scheduled, and manual runs remain broad because they protect branch
health, release readiness, and nightly confidence rather than one PR diff.
Quality contract:
- Any path-gated lane must be forced by `ci:full`.
- Every expensive lane needs a matching force label so maintainers can request
coverage without pushing a no-op commit.
- Workflow, shared setup, toolchain, lockfile, and classifier changes should run
the affected expensive lanes because they can change CI behavior even when
product code did not move.
- The `Tests` workflow runs the classifier self-test before consuming classifier
outputs. That self-test covers representative path matches and label forcing
so a future edit cannot silently weaken the broadest PR test gate.
- When splitting a long lane, keep the same substantive commands unless the PR
explicitly documents the safety reason for removing one.
Long deterministic E2E gates are split into named parallel slices for unit/UI
coverage, browser coverage, diagnostics, and scenario execution. The visible
`Zero-Key Deterministic E2E` check is an aggregate status over those slices, so
reviewers can see the failing surface without opening one giant serial log.
Plugin tests are also split across `TEST_SHARD=1/4` through `4/4` in the
`Tests` workflow. The root `test:plugins` script uses the cross-package runner
so shard membership is deterministic by package path, while the visible
`Plugin Tests` check remains an aggregate over the shard matrix.
Why the aggregate stays:
- Branch protection and reviewer muscle memory can keep using one stable check.
- The underlying slices can run in parallel and fail with precise names.
- Manual review becomes easier because a browser failure, diagnostics failure,
or scenario-runner failure points at the relevant log immediately.
Related CI docs:
- `CHANGELOG.md` records workflow policy changes and the reason they happened.
- `ROADMAP.md` tracks future CI performance work that should preserve gate
quality.
### Main CI (`ci.yaml`)
Runs on PRs and pushes to main:
- Typecheck + core/plugin tests
- Linting and formatting checks
- Build verification
- Dev startup + HMR propagation
- Interop TypeScript tests (`packages/interop`)
The broader `test.yml` orchestrator runs automatically on `develop` only to
avoid duplicating the main-branch CI gate. Secret-free deterministic zero-key
coverage for PRs to either protected branch is handled by `scenario-pr.yml`;
`test.yml` keeps the broader develop push/PR, manual, and scheduled coverage.
### Live E2E
PR E2E does not require `CEREBRAS_API_KEY`, `OPENAI_API_KEY`, or any other paid
provider key. Live/provider-key coverage belongs to the dedicated live jobs and
workflows (`cloud-live-e2e`, `provider-live-e2e`, `live-scenarios.yml`,
`scenario-matrix.yml`) where missing-key behavior is documented per lane.
## Code Review Workflows
### Claude Code Review (`claude-code-review.yml`)
Automated PR review using Claude. Checks for:
- Security issues (hardcoded keys, SQL injection, XSS)
- Test coverage
- TypeScript types (no `any`)
- Correct tooling (bun, vitest)
### Claude Security Review (`claude-security-review.yml`)
Dedicated security-focused review for code changes.
### Claude Interactive (`claude.yml`)
Responds to `@claude` mentions in issues and PRs.
## Documentation Workflows
### Docs CI (`docs-ci.yml`)
Documentation quality workflow:
- **Dead Link Checking:** Scans for broken internal/external links
- **Quality Checks:** Double headers, missing frontmatter, heading hierarchy
Automatically creates PRs with fixes when issues are found.
### JSDoc Automation (`jsdoc-automation.yml`)
Manual workflow for generating JSDoc documentation.
## Manual Release Process
### 1. Create a GitHub Release
1. Go to Releases → Create new release
2. Create a new tag: `v2.0.0` (follows semver)
3. Add release notes
4. Publish release
### 2. Automated Publishing
The release will trigger:
- `release.yaml` → NPM packages
### 3. Manual publishing
Use `bunx lerna publish` from the repo root when automation is not sufficient (see `release.yaml`).
## Setting Up Secrets
### Required Secrets
| Secret | Purpose | How to Get |
|--------|---------|------------|
| `NPM_TOKEN` | NPM publishing | [npmjs.com/settings/~/tokens](https://www.npmjs.com/settings/~/tokens) |
| `ANTHROPIC_API_KEY` | Claude workflows | [console.anthropic.com](https://console.anthropic.com) |
| `OPENAI_API_KEY` | Opt-in live/provider-key lanes | [platform.openai.com](https://platform.openai.com) |
### Optional Secrets
| Secret | Purpose |
|--------|---------|
| `PHALA_CLOUD_API_KEY` | TEE deployment |
| `GH_PAT` | Cross-repo operations |
Turbo caching is GitHub-native (`.github/actions/turbo-cache-github` via
`setup-bun-workspace`) — no Vercel SaaS remote cache, so `TURBO_TOKEN` /
`TURBO_TEAM` are no longer used and are banned by
`ci-workflow-dedup-contract.mjs` (#12341).
## Package dependencies
NPM packages are ordered by the monorepo graph; `release.yaml` / Lerna handle publish ordering for `@elizaos/*` packages.
## Troubleshooting
### CI Failures
1. Check if tests pass locally: `bun run test`
2. Check formatting: `bun run format:check`
3. Check linting: `bun run lint`
### Release Failures
1. Verify secrets are configured
2. Check workflow logs for specific errors
3. For NPM: ensure package versions are unique
### Claude Workflow Issues
1. Verify `ANTHROPIC_API_KEY` is set
2. Check rate limits on Anthropic API
3. Review Claude's output in workflow logs
+79
View File
@@ -0,0 +1,79 @@
# CI Workflow Roadmap
This roadmap is for making CI faster and friendlier without weakening the
quality gate.
## Principles
- Prefer affected-surface execution over blanket PR execution.
Why: contributors should not wait on unrelated platform builds, but affected
code still needs the same depth of verification.
- Keep broad verification on push, schedule, and manual dispatch.
Why: selective PR checks optimize review latency; broad branch checks catch
integration issues that only appear after many changes compose.
- Make every skip explainable and overridable.
Why: maintainers need confidence that a skipped lane was intentionally out of
scope, and they need a label escape hatch when judgment says to run it anyway.
- Split long serial lanes before removing coverage.
Why: parallelism usually improves developer experience without trading away
signal.
## Near Term
- Record workflow-level and job-level duration trends for PR and `develop`
pushes.
Why: the current long pole can move as workflows are split. Duration data lets
us optimize the real bottleneck instead of guessing from one run.
- Continue splitting long jobs into independently named slices with aggregate
status checks.
Why: aggregate checks preserve branch-protection simplicity while named slices
improve failure triage and parallelism.
- Expand classifier self-tests when adding a new path-gated workflow.
Why: every new gate adds skip logic. The test suite should grow with the blast
radius of that logic.
## Next
- Move repeated setup into reusable actions where it does not hide important
workflow behavior.
Why: duplicated setup makes split jobs noisy, but overly opaque setup makes CI
harder to debug. Reusable actions should reduce repetition while keeping logs
readable.
- Add a documented owner map for path-gate rules.
Why: package owners can judge whether a path belongs in server, client,
plugin, cloud, mobile, desktop, Docker, or E2E coverage better than a generic
workflow edit can.
- Evaluate matrix shaping for Windows CI.
Why: recent completed `develop` runs show Windows CI as the workflow-level
long pole. More parallelism or tighter affected-surface routing there may
produce the biggest wall-clock win.
## Later
- Publish a compact CI timing summary on PRs.
Why: contributors should see whether the slowest check is queue time, setup,
test execution, artifact upload, or a platform-specific lane.
- Add a periodic full-gate audit that compares path-gated PR behavior against a
broad run for representative diffs.
Why: selective CI needs periodic calibration so the faster path remains as
trustworthy as the older all-in path.
@@ -0,0 +1,412 @@
name: Actions Zombie Janitor
# Reap zombie workflow runs (dead-runner `in_progress`, wedged `queued`,
# production deploy `pending` with zero jobs, stale `waiting`) that freeze org
# concurrency or silently block deploys.
#
# WHY: when a self-hosted runner box dies, its runs stay `in_progress` until
# the job timeout (6h default, up to 12h here) — and each one keeps holding
# org concurrency slots. With enough zombies, NO new job can start repo-wide,
# including GitHub-hosted ones (observed 2026-07-01 ~21:3322:30 UTC: ~40
# zombies from a 17:4x runner death froze every queue — PR checks, gitleaks,
# and the production deploy dispatches; see #10839 updates 912 and #11045).
# Cancelling QUEUED runs does not free slots — the slots are held by the dead
# in-progress runs — but a WEDGED queued run is still poison: it holds its
# concurrency group and GitHub replaces-and-cancels every newer pending run in
# that group.
#
# WHY `waiting` RUNS TOO (2026-07-06 incident, #15147): a run `waiting` on an
# environment approval that never comes holds its concurrency group until
# GitHub's 30-day auto-expiry — with `cancel-in-progress: false` every later
# run of that group sits `pending` with zero jobs and dies silently. Two runs
# waiting since 2026-06-18 deadlocked the production backend + provisioning
# worker deploy lanes for 18 days; production ran June-18 code the whole time.
# Nobody approves a production deploy two days late — reaping the gate frees
# the group so the next dispatch deploys tip.
#
# WHY THE JANITOR ITSELF MUST BE FREEZE-PROOF (2026-07-02/03 incident): the
# previous single-lane design (one ubuntu-latest job + one shared concurrency
# group) had zero successful executions from 2026-07-02 14:15 UTC through
# 2026-07-03 while a fresh zombie batch (born 07-02 23:3123:50, hetzner-robot
# death) froze the queues ~3x in 24h and ~34 runs had to be force-cancelled BY
# HAND. Two failure modes compounded:
# 1. SLOT STARVATION — the janitor needs a runner slot from the very pool
# the zombies exhaust, so during a full freeze its scheduled runs starve
# in `queued` and never execute.
# 2. CONCURRENCY-GROUP SELF-BRICK — one wedged queued janitor run (e.g.
# 28635685728, live-verified `queued` for 13+ hours while other
# ubuntu-latest jobs started fine) held the shared group forever; GitHub
# then replaced-and-cancelled every newer pending tick (10 consecutive
# janitor runs cancelled unexecuted, 28628876713…28666171315).
# Fixes here:
# - TWO INDEPENDENT LANES (matrix): a GitHub-hosted job and a self-hosted
# robot-fleet job. A freeze that exhausts one pool leaves the other lane
# runnable; either lane alone fully reaps. Lanes never touch each other.
# - NO CONCURRENCY GROUP. Overlapping janitors are harmless (cancels are
# idempotent; every candidate is re-verified live), and any queued backlog
# purges itself: each lane cancels superseded queued janitor ticks.
# - Wedged-queued reaping: own ticks queued >60 min, anything queued >12 h.
# - Production Cloud CF Deploy pending/zero-job reaping: a prod deploy that
# never instantiates a job is neither an approval gate nor a runner backlog;
# after 15 minutes the janitor force-cancels it and redispatches once
# (#15503/#15499).
# - actions/github-script instead of gh/jq — self-hosted boxes only need
# the runner-bundled Node.
#
# WHAT COUNTS AS A ZOMBIE `in_progress` RUN — both must hold:
# 1. the current attempt is older than MAX_AGE_MINUTES (default 120), AND
# 2. no job OR STEP of the run started or completed within IDLE_MINUTES
# (default 90).
# The idle check is load-bearing: right after a freeze clears, hours-old runs
# legitimately start executing their backlogged jobs — age alone would kill
# healthy recovering runs; age+idle reaps only runs nothing is working on.
# Step timestamps (new) keep long multi-step jobs alive while their steps
# progress; a dead runner freezes job AND step timestamps, so zombies still
# trip the check. IDLE_MINUTES must stay above the longest legitimately-silent
# single-step stretch; 90 min clears every current suite. The two 12h KVM/AOSP
# suites run single steps silent for longer than any sane threshold, so they
# are exempt by name (their zombies fall to their own job timeout).
#
# Tuning (repo variables, no code change needed):
# ACTIONS_JANITOR_DISABLED=true — kill switch (both lanes)
# ACTIONS_JANITOR_ROBOT_LANE_DISABLED — =true remaps the robot lane onto
# ubuntu-latest (a harmless idempotent duplicate; `matrix` is not a legal
# context in a job-level `if`, so the lane is disabled by re-homing it)
# ACTIONS_JANITOR_ROBOT_RUNNER_JSON — robot lane runs-on JSON array
# ACTIONS_JANITOR_MAX_AGE_MINUTES — default 120
# (legacy ACTIONS_JANITOR_MAX_AGE_HOURS honored if the minutes var is unset)
# ACTIONS_JANITOR_IDLE_MINUTES — default 90
# ACTIONS_JANITOR_QUEUED_MAX_AGE_HOURS — default 24 (wedged-queued reap;
# deliberately at GitHub's own queued-job expiry so the sweep only catches
# runs GitHub failed to fail — hours-old queued runs can be legitimate
# freeze backlog that still executes once slots free, and cancelling a
# required PR check strands the PR red; tune down mid-incident if needed)
# ACTIONS_JANITOR_PENDING_ZERO_JOB_MINUTES — default 15 (production Cloud CF
# Deploy pending/zero-job zombie reap + one redispatch)
# ACTIONS_JANITOR_SELF_QUEUED_MINUTES — default 60 (own stale ticks)
# ACTIONS_JANITOR_WAITING_MAX_AGE_HOURS — default 48 (stale environment-gate
# reap; far above any sane approval latency, far below GitHub's 30-day
# expiry that let #15147 fester)
# ACTIONS_JANITOR_EXEMPT_WORKFLOWS — comma-separated workflow names
on:
schedule:
- cron: "*/15 * * * *"
workflow_dispatch:
inputs:
dry_run:
description: Log what would be cancelled without cancelling
required: false
default: false
type: boolean
permissions:
actions: write
jobs:
reap:
name: Reap zombie runs (${{ matrix.lane }})
strategy:
fail-fast: false
matrix:
include:
- lane: github-hosted
runner: '["ubuntu-latest"]'
- lane: robot-fleet
runner: ${{ vars.ACTIONS_JANITOR_ROBOT_LANE_DISABLED == 'true' && '["ubuntu-latest"]' || vars.ACTIONS_JANITOR_ROBOT_RUNNER_JSON || '["self-hosted","Linux","X64","hetzner-robot"]' }}
runs-on: ${{ fromJSON(matrix.runner) }}
if: github.repository == 'elizaOS/eliza' && vars.ACTIONS_JANITOR_DISABLED != 'true'
timeout-minutes: 10
steps:
- name: Cancel zombie in_progress + wedged queued + pending deploy + stale waiting runs
uses: actions/github-script@v7
env:
MAX_AGE_MINUTES: ${{ vars.ACTIONS_JANITOR_MAX_AGE_MINUTES || '' }}
LEGACY_MAX_AGE_HOURS: ${{ vars.ACTIONS_JANITOR_MAX_AGE_HOURS || '' }}
IDLE_MINUTES: ${{ vars.ACTIONS_JANITOR_IDLE_MINUTES || '90' }}
QUEUED_MAX_AGE_HOURS: ${{ vars.ACTIONS_JANITOR_QUEUED_MAX_AGE_HOURS || '24' }}
PENDING_ZERO_JOB_MINUTES: ${{ vars.ACTIONS_JANITOR_PENDING_ZERO_JOB_MINUTES || '15' }}
SELF_QUEUED_MINUTES: ${{ vars.ACTIONS_JANITOR_SELF_QUEUED_MINUTES || '60' }}
WAITING_MAX_AGE_HOURS: ${{ vars.ACTIONS_JANITOR_WAITING_MAX_AGE_HOURS || '48' }}
EXEMPT_WORKFLOWS: ${{ vars.ACTIONS_JANITOR_EXEMPT_WORKFLOWS || 'ElizaOS Cuttlefish,ElizaOS OpenAgent E1 (RISC-V AI SoC)' }}
DRY_RUN: ${{ inputs.dry_run == true && 'true' || 'false' }}
LANE: ${{ matrix.lane }}
with:
script: |
const cfg = {
maxAgeMin:
Number(process.env.MAX_AGE_MINUTES) ||
Number(process.env.LEGACY_MAX_AGE_HOURS) * 60 ||
120,
idleMin: Number(process.env.IDLE_MINUTES) || 90,
queuedMaxAgeMin: (Number(process.env.QUEUED_MAX_AGE_HOURS) || 24) * 60,
pendingZeroJobMin: Number(process.env.PENDING_ZERO_JOB_MINUTES) || 15,
selfQueuedMin: Number(process.env.SELF_QUEUED_MINUTES) || 60,
waitingMaxAgeMin: (Number(process.env.WAITING_MAX_AGE_HOURS) || 48) * 60,
// Janitor lane jobs have timeout-minutes:10; an own-workflow run
// in_progress far past that means its runner died mid-reap.
selfInProgressMin: 60,
exempt: (process.env.EXEMPT_WORKFLOWS || '')
.split(',')
.map((s) => s.trim())
.filter(Boolean),
dryRun: process.env.DRY_RUN === 'true',
lane: process.env.LANE || 'unknown',
};
const { owner, repo } = context.repo;
const now = Date.now();
const minutesSince = (iso) => Math.floor((now - Date.parse(iso)) / 60000);
const rows = [];
const reap = async (run, ageMin, idleMin, why) => {
const label = `${run.id} ${run.name} (age ${ageMin}m, idle ${idleMin}m, ${why})`;
if (cfg.dryRun) {
core.info(`would cancel: ${label}`);
rows.push([`${run.id}`, run.name, `${ageMin}`, `${idleMin}`, `would cancel (dry run) — ${why}`]);
return;
}
// FORCE-cancel: a plain cancel silently fails to reap runs whose
// runner died (they sit in cancel-requested limbo until the job
// timeout — exactly the runs this janitor exists for). The
// force-cancel endpoint kills them immediately; fall back to the
// polite cancel for anything the force path rejects.
try {
await github.request('POST /repos/{owner}/{repo}/actions/runs/{run_id}/force-cancel', {
owner, repo, run_id: run.id,
});
core.info(`force-cancelled: ${label}`);
rows.push([`${run.id}`, run.name, `${ageMin}`, `${idleMin}`, `force-cancelled — ${why}`]);
} catch (e) {
try {
await github.rest.actions.cancelWorkflowRun({ owner, repo, run_id: run.id });
core.info(`cancelled: ${label}`);
rows.push([`${run.id}`, run.name, `${ageMin}`, `${idleMin}`, `cancelled — ${why}`]);
} catch (e2) {
core.info(`cancel failed (likely already completed / raced with sibling lane): ${label}: ${e2.status ?? e2.message}`);
}
}
};
// The list endpoint serves STALE entries for force-killed runs —
// re-verify each candidate individually before judging it.
const liveStatus = async (runId) => {
try {
return (await github.rest.actions.getWorkflowRun({ owner, repo, run_id: runId })).data.status;
} catch {
return null;
}
};
const liveRun = async (runId) => {
try {
return (await github.rest.actions.getWorkflowRun({ owner, repo, run_id: runId })).data;
} catch {
return null;
}
};
// Both listings are pre-filtered server-side to runs CREATED before
// the smallest relevant threshold: the previous `--limit 100`
// newest-first listing could drop the oldest zombies — the exact
// runs to reap — once the queue saturated (observed: >1000 queued
// runs during the 2026-07-03 freeze put the 13h-wedged run beyond
// any sane pagination depth). A run created recently cannot have
// started earlier, so nothing eligible is ever excluded; re-run
// attempts (old created_at, fresh run_started_at) stay listed and
// are age-gated per attempt below.
const createdBefore = (minutes) =>
`<${new Date(now - minutes * 60000).toISOString()}`;
// ---- 1. zombie in_progress runs (these hold the concurrency slots) ----
const inProgress = await github.paginate(github.rest.actions.listWorkflowRunsForRepo, {
owner, repo, status: 'in_progress', per_page: 100,
created: createdBefore(Math.min(cfg.maxAgeMin, cfg.selfInProgressMin)),
});
inProgress.sort(
(a, b) => Date.parse(a.run_started_at || a.created_at) - Date.parse(b.run_started_at || b.created_at),
);
core.info(`[${cfg.lane}] in_progress listed: ${inProgress.length}`);
for (const run of inProgress) {
if (run.id === context.runId) continue;
// Age of the CURRENT attempt (run_started_at resets on re-run);
// created_at would overstate the age of re-run attempts.
const ageMin = minutesSince(run.run_started_at || run.created_at);
if (run.name === context.workflow) {
// Never touch the live sibling lane; only reap a janitor run
// whose own runner died mid-reap.
if (ageMin < cfg.selfInProgressMin) continue;
if ((await liveStatus(run.id)) !== 'in_progress') continue;
await reap(run, ageMin, ageMin, 'dead janitor run');
continue;
}
if (cfg.exempt.includes(run.name)) {
core.info(`exempt: ${run.id} ${run.name}`);
continue;
}
if (ageMin < cfg.maxAgeMin) continue;
const status = await liveStatus(run.id);
if (status !== 'in_progress') {
core.info(`skip (list-stale, actually '${status}'): ${run.id} ${run.name}`);
continue;
}
// Newest job OR step activity across up to 100 jobs; a run with
// no timestamps at all idles from its attempt start.
let latest = 0;
try {
const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, {
owner, repo, run_id: run.id, filter: 'latest', per_page: 100,
});
for (const j of jobs) {
for (const t of [j.started_at, j.completed_at]) {
if (t) latest = Math.max(latest, Date.parse(t));
}
for (const s of j.steps ?? []) {
for (const t of [s.started_at, s.completed_at]) {
if (t) latest = Math.max(latest, Date.parse(t));
}
}
}
} catch (e) {
core.warning(`jobs fetch failed for ${run.id}: ${e.status ?? e.message}`);
}
const idleMin = latest ? Math.floor((now - latest) / 60000) : ageMin;
if (idleMin < cfg.idleMin) {
core.info(`alive: ${run.id} ${run.name} (age ${ageMin}m, idle ${idleMin}m)`);
continue;
}
await reap(run, ageMin, idleMin, 'zombie in_progress');
}
// ---- 2. wedged queued runs (these brick concurrency groups) ----
// Cancelling queued runs does not free slots, but a run stuck in
// `queued` for hours holds its concurrency group and gets every
// newer pending run of that group replaced-and-cancelled (this is
// what bricked the janitor itself on 2026-07-03). Own ticks are
// superseded after selfQueuedMin (a later tick reaps strictly
// better). Other workflows only past queuedMaxAgeMin (24h —
// GitHub's own queued expiry): an hours-old queued run can be
// legitimate freeze backlog that still executes once slots free,
// and cancelling a required PR check strands the PR red.
// Two tightly-scoped listings instead of one walk of the whole
// queued backlog (which exceeds 1000 runs mid-freeze and would eat
// the API budget): own ticks via the workflow-scoped endpoint, and
// a repo-wide sweep only past the (12h) wedge threshold.
const selfQueued = await github.paginate(github.rest.actions.listWorkflowRuns, {
owner, repo, workflow_id: 'actions-zombie-janitor.yml', status: 'queued', per_page: 100,
created: createdBefore(cfg.selfQueuedMin),
});
const wedgedQueued = await github.paginate(github.rest.actions.listWorkflowRunsForRepo, {
owner, repo, status: 'queued', per_page: 100,
created: createdBefore(cfg.queuedMaxAgeMin),
});
const queued = new Map();
for (const run of [...selfQueued, ...wedgedQueued]) queued.set(run.id, run);
core.info(`[${cfg.lane}] stale queued listed: ${queued.size} (self ${selfQueued.length}, wedged ${wedgedQueued.length})`);
for (const run of queued.values()) {
if (run.id === context.runId) continue;
const ageMin = minutesSince(run.created_at);
const isSelf = run.name === context.workflow;
if (ageMin < (isSelf ? cfg.selfQueuedMin : cfg.queuedMaxAgeMin)) continue;
if ((await liveStatus(run.id)) !== 'queued') continue;
await reap(run, ageMin, ageMin, isSelf ? 'superseded janitor tick' : 'wedged queued');
}
// ---- 3. production deploy pending/zero-job zombies ----
// These runs never instantiate a job, so no environment approval,
// runner assignment, or job-level concurrency record exists for a
// human to inspect. Scope this to main->production Cloud CF Deploy:
// staging can legitimately sit behind active deploy work, while a
// production run with zero jobs and no active same-branch sibling is
// silent release blockage (#15503/#15499).
const activeCloudDeploy = new Map();
for (const status of ['in_progress', 'waiting']) {
const activeRuns = await github.paginate(github.rest.actions.listWorkflowRuns, {
owner, repo, workflow_id: 'cloud-cf-deploy.yml', status, per_page: 100,
});
for (const run of activeRuns) {
activeCloudDeploy.set(`${run.head_branch || ''}:${status}`, run);
}
}
const pendingCloudDeploy = await github.paginate(github.rest.actions.listWorkflowRuns, {
owner, repo, workflow_id: 'cloud-cf-deploy.yml', status: 'pending', per_page: 100,
created: createdBefore(cfg.pendingZeroJobMin),
});
core.info(`[${cfg.lane}] pending Cloud CF Deploy listed: ${pendingCloudDeploy.length}`);
for (const run of pendingCloudDeploy) {
if (run.id === context.runId) continue;
if (run.head_branch !== 'main') continue;
if (activeCloudDeploy.has('main:in_progress') || activeCloudDeploy.has('main:waiting')) {
core.info(`skip pending prod deploy ${run.id}: active same-branch deploy exists`);
continue;
}
const ageMin = minutesSince(run.created_at);
if (ageMin < cfg.pendingZeroJobMin) continue;
const live = await liveRun(run.id);
if (live?.status !== 'pending') {
core.info(`skip (list-stale, actually '${live?.status ?? 'missing'}'): ${run.id} ${run.name}`);
continue;
}
const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, {
owner, repo, run_id: run.id, filter: 'latest', per_page: 100,
});
if (jobs.length > 0) {
core.info(`skip pending prod deploy ${run.id}: ${jobs.length} job(s) exist`);
continue;
}
await reap(run, ageMin, ageMin, 'production deploy pending with zero jobs');
if (run.actor?.login === 'github-actions[bot]') {
core.warning(`cancelled bot-dispatched pending prod deploy ${run.id}; not redispatching again`);
continue;
}
if (cfg.dryRun) {
core.info(`would redispatch Cloud CF Deploy production at ${run.head_branch}`);
rows.push([`${run.id}`, run.name, `${ageMin}`, `${ageMin}`, 'would redispatch production deploy (dry run)']);
continue;
}
await github.rest.actions.createWorkflowDispatch({
owner, repo, workflow_id: 'cloud-cf-deploy.yml', ref: 'main',
inputs: { environment: 'production', force: 'false' },
});
core.warning(`redispatched Cloud CF Deploy production after cancelling pending/zero-job run ${run.id}`);
rows.push([`${run.id}`, run.name, `${ageMin}`, `${ageMin}`, 'redispatched production deploy once']);
}
// ---- 4. stale waiting runs (unapproved environment gates) ----
// A run `waiting` on an environment approval that never comes
// holds its concurrency group until GitHub's 30-day auto-expiry;
// with cancel-in-progress:false every later run of the group sits
// `pending` with zero jobs and dies silently (#15147: 18-day
// deadlock of both production deploy lanes). 48h is far above any
// sane approval latency — reap the gate so the group frees.
const staleWaiting = await github.paginate(github.rest.actions.listWorkflowRunsForRepo, {
owner, repo, status: 'waiting', per_page: 100,
created: createdBefore(cfg.waitingMaxAgeMin),
});
core.info(`[${cfg.lane}] stale waiting listed: ${staleWaiting.length}`);
for (const run of staleWaiting) {
// Attempt age: a re-run of an old run keeps created_at (which
// the server-side filter uses, as a superset) but resets
// run_started_at — gate on the current attempt.
const ageMin = minutesSince(run.run_started_at || run.created_at);
if (ageMin < cfg.waitingMaxAgeMin) continue;
if ((await liveStatus(run.id)) !== 'waiting') continue;
await reap(run, ageMin, ageMin, 'stale waiting (unapproved environment gate)');
}
// ---- summary ----
core.summary.addHeading(`Zombie janitor [${cfg.lane}] — ${new Date(now).toISOString()}`, 2);
if (rows.length === 0) {
core.summary.addRaw('No zombies found.', true);
} else {
core.summary.addTable([
[
{ data: 'run', header: true },
{ data: 'workflow', header: true },
{ data: 'age (min)', header: true },
{ data: 'idle (min)', header: true },
{ data: 'action', header: true },
],
...rows,
]);
}
await core.summary.write();
+548
View File
@@ -0,0 +1,548 @@
name: android-device-e2e
# Real-device Android e2e: builds the WebView-debuggable APK, boots an emulator,
# and drives the real app via Playwright's Android driver + the on-device agent
# smoke. A weekly host-backed schedule gives the lane unattended cadence without
# burning an emulator on every PR; workflow_dispatch still exposes the full
# backend selector for manual cloud/local investigations.
#
# IMPORTANT coverage note: the embedded on-device agent (bun + llama) runs on
# real arm64 hardware but SIGSEGVs on a stock x86_64 emulator, so the LOCAL
# route's full chat round-trip only goes green on a self-hosted arm64 device
# runner. On a GitHub-hosted x86_64 emulator this lane proves: build + install +
# WebView attach + route navigation, and runs the local smoke as a (currently
# expected-to-surface) signal. For green route coverage on the emulator, supply
# an Eliza Cloud token (ELIZA_CLOUD_AUTH_TOKEN secret) so the app onboards to a
# real cloud agent.
#
# iOS onboarding-smoke parity (#13580): mobile-build-smoke.yml runs
# ios-onboarding-smoke.mjs on every path-gated PR, but the Android
# onboarding->home smoke (test:e2e:android:onboarding) is intentionally NOT
# per-PR. The iOS smoke boots a simulator on a macOS runner cheaply; the
# Android smoke needs a KVM-accelerated x86_64 emulator on a Linux runner,
# which is too heavy to boot on every PR (issue #9943). Android
# onboarding->home instead runs on three cadences: the label-gated
# (ci:device) pr-device-smoke job below, the weekly host-backed schedule,
# and workflow_dispatch -- so an onboarding regression is caught on a PR
# opt-in or within one weekly cycle rather than never. This is the
# documented divergence the parity checkbox in #13580 asks for.
on:
schedule:
# Weekly hosted x86_64 emulator run, backend=host by default below. The
# self-hosted arm64 LOCAL route remains separate because bun+llama SIGSEGVs
# on the stock hosted x86_64 emulator (see coverage note above).
- cron: "17 9 * * 1"
workflow_dispatch:
inputs:
backend:
description: "WebView backend for route coverage (cloud|host|local)"
type: choice
options: [cloud, host, local]
default: cloud
api-level:
description: "Android emulator API level"
default: "34"
# Reachable on normal develop PRs and release-promotion PRs, but only when
# explicitly opted in with the `ci:device` label (issue #9943) — booting an
# emulator is slow/heavy, so we never burn a runner on every PR. The
# pr-device-smoke job below is gated on that label and runs a MINIMAL slice
# (build + install + WebView attach + onboarding→home + one host-backed chat
# turn); the full dispatch matrix above is unchanged.
pull_request:
branches: [develop, main]
types: [opened, synchronize, reopened, labeled]
concurrency:
group: android-device-e2e-${{ github.ref }}
cancel-in-progress: true
jobs:
android-e2e:
# Full lane runs on manual dispatch and on the weekly scheduled host-backed
# cadence. Never on PR; PRs use the label-gated minimal smoke below.
if: github.event_name == 'workflow_dispatch' || github.event_name == 'schedule'
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 90
permissions:
contents: read
env:
ELIZA_MOBILE_REPO_ROOT: ${{ github.workspace }}
ELIZA_WEBVIEW_DEBUG: "1"
ELIZA_BUN_RISCV64_OPTIONAL: "1"
ELIZA_ANDROID_SKIP_FORK_LLAMA_LIB: "1"
ELIZA_ANDROID_BACKEND: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.backend || 'host' }}
ELIZA_CLOUD_AUTH_TOKEN: ${{ secrets.ELIZA_CLOUD_AUTH_TOKEN }}
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.14" # pinned: repo packageManager is bun@1.4.0, an unpublished version (GH+npm 404) that makes bare setup-bun fail here; match .github/ci-bun-version.json (#15071)
- uses: actions/setup-java@v5
with:
distribution: temurin
java-version: "21"
- uses: android-actions/setup-android@v4
- name: Install workspace
run: bun install
- name: Build WebView-debuggable APK
run: bun run --cwd packages/app build:android
# KVM enables a usable emulator on GitHub ubuntu runners.
- name: Enable KVM
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
| sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules && sudo udevadm trigger --name-match=kvm
- name: Route coverage on emulator
uses: reactivecircus/android-emulator-runner@v2
with:
api-level: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.api-level || '34' }}
arch: x86_64
ram-size: 6144M
# avx2/fma passthrough so the (x86) on-device agent doesn't SIGILL.
emulator-options: -no-window -no-snapshot -no-boot-anim -gpu swiftshader_indirect -qemu -cpu qemu64,+avx,+avx2,+f16c,+fma
script: |
set -euo pipefail
adb root || true
adb shell setenforce 0 || true
bun run --cwd packages/app build # ensure web dist
mkdir -p packages/app/test-results/android-onboarding-to-home
ELIZA_API_PORT=31337 \
ELIZA_PAIRING_DISABLED=1 \
node packages/app-core/scripts/run-node-tsx.mjs \
packages/app-core/scripts/serve-real-local-agent.ts \
> packages/app/test-results/android-onboarding-to-home/host-agent.log 2>&1 &
HOST_AGENT_PID=$!
trap 'kill "$HOST_AGENT_PID" 2>/dev/null || true' EXIT
for i in $(seq 1 90); do
if curl -fsS \
-H 'X-ElizaOS-Client-Id: android-onboarding-ci' \
http://127.0.0.1:31337/api/health >/tmp/android-host-agent-health.json; then
cat /tmp/android-host-agent-health.json
break
fi
if ! kill -0 "$HOST_AGENT_PID" 2>/dev/null; then
echo "Host agent exited before becoming healthy"
cat packages/app/test-results/android-onboarding-to-home/host-agent.log
exit 1
fi
sleep 2
done
curl -fsS \
-H 'X-ElizaOS-Client-Id: android-onboarding-ci' \
http://127.0.0.1:31337/api/health >/dev/null
ELIZA_ANDROID_BACKEND=host \
ELIZA_ANDROID_REQUIRE_AGENT=1 \
bun run --cwd packages/app test:e2e:android:onboarding
if [ "${ELIZA_ANDROID_BACKEND}" = "local" ]; then
# LOCAL route is still signal-only on hosted x86_64: the embedded
# bun+llama agent is known to require a self-hosted arm64 device
# runner. Keep surfacing the artifact, but do not make the hosted
# emulator falsely red for that known platform gap.
ELIZA_ANDROID_REQUIRE_AGENT=1 \
bun run --cwd packages/app test:e2e:android:routes || true
else
# Host/cloud route coverage is the scheduled unattended health
# signal. Fail red here instead of hiding regressions behind a
# permanent signal-only `|| true` leg.
ELIZA_ANDROID_REQUIRE_AGENT=0 \
bun run --cwd packages/app test:e2e:android:routes
fi
ELIZA_ANDROID_BACKEND=host \
ELIZA_ANDROID_REQUIRE_AGENT=1 \
bun run --cwd packages/app test:e2e:android:native-plugin-view
# Local on-device agent smoke (surfaces failure loudly; expected to
# fail on stock x86 emulator — see coverage note above).
bun run --cwd packages/app test:sim:local-chat:android:live || true
# Seeded launcher gesture loop (#12377): ≥200 real device swipes/taps
# with per-action rail invariants (data-page + AX probe + inertness +
# focus), chunked screenrecord + logcat. Signal-only on the x86
# emulator until proven emulator-green; the seed prints for replay.
ELIZA_ANDROID_BACKEND=host \
ELIZA_ANDROID_REQUIRE_AGENT=1 \
bun run --cwd packages/app test:e2e:android:launcher-loop || true
- name: Cloud provisioning probe
run: |
if [ -z "${ELIZA_CLOUD_AUTH_TOKEN:-}" ]; then
{
echo "### Cloud provisioning probe"
echo ""
echo "skipped: missing ELIZA_CLOUD_AUTH_TOKEN"
} >> "$GITHUB_STEP_SUMMARY"
echo "Cloud provisioning probe skipped: missing ELIZA_CLOUD_AUTH_TOKEN"
exit 0
fi
bun run --cwd packages/app test:e2e:android:cloud
- uses: actions/upload-artifact@v7
if: always()
with:
name: android-e2e-artifacts
path: |
packages/app/test-results/**
packages/app/playwright-report/**
if-no-files-found: ignore
notify-scheduled-failure:
needs: android-e2e
if: ${{ always() && github.event_name == 'schedule' && needs.android-e2e.result != 'success' }}
runs-on: ubuntu-24.04
permissions:
contents: read
issues: write
env:
ANDROID_E2E_RESULT: ${{ needs.android-e2e.result }}
ELIZA_ANDROID_BACKEND: host
steps:
- name: Open / update scheduled failure issue
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3
with:
script: |
const title = "Scheduled Android device e2e is failing (#13580)";
const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
const artifactsUrl = `${runUrl}#artifacts`;
const runDate = new Date().toISOString();
const result = process.env.ANDROID_E2E_RESULT || "unknown";
const body = [
"The weekly host-backed Android device e2e cadence failed.",
"",
`Run: ${runUrl}`,
`Artifacts: ${artifactsUrl}`,
`Workflow: \`${context.workflow}\``,
`Android job result: \`${result}\``,
`Backend: \`${process.env.ELIZA_ANDROID_BACKEND || "host"}\``,
`Observed: ${runDate}`,
"",
"This issue is auto-maintained by `.github/workflows/android-device-e2e.yml` so scheduled failures are visible outside Actions history.",
"",
"Known #13580 residuals still tracked separately:",
"- LOCAL arm64 on-device route needs a self-hosted arm64 runner.",
"- Signal-only PR legs should be promoted only after emulator-green evidence exists.",
].join("\n");
const openIssues = await github.paginate(github.rest.issues.listForRepo, {
owner: context.repo.owner,
repo: context.repo.repo,
state: "open",
per_page: 100,
});
const existing = openIssues.find((issue) => !issue.pull_request && issue.title === title);
if (existing) {
await github.rest.issues.update({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: existing.number,
body,
});
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: existing.number,
body: `Scheduled Android device e2e failed again (${result}): ${runUrl}`,
});
core.info(`Updated scheduled failure issue #${existing.number}`);
} else {
const { data: created } = await github.rest.issues.create({
owner: context.repo.owner,
repo: context.repo.repo,
title,
body,
});
core.info(`Opened scheduled failure issue #${created.number}`);
}
# Minimal, label-gated PR device smoke (issue #9943). Reachable on a PR only
# when the `ci:device` label is applied, so it never burns an emulator runner
# on every PR. Runs the proven-green x86 slice: build the WebView-debuggable
# APK, boot the emulator, attach, and drive onboarding→home plus a chat turn
# against the host-run agent (the host backend runs the agent on the runner, so
# it works on a stock x86 emulator where the embedded on-device agent SIGSEGVs).
pr-device-smoke:
if: ${{ github.event_name == 'pull_request' && contains(github.event.pull_request.labels.*.name, 'ci:device') }}
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 60
env:
ELIZA_MOBILE_REPO_ROOT: ${{ github.workspace }}
ELIZA_WEBVIEW_DEBUG: "1"
ELIZA_BUN_RISCV64_OPTIONAL: "1"
ELIZA_ANDROID_SKIP_FORK_LLAMA_LIB: "1"
ELIZA_ANDROID_BACKEND: "host"
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.14" # pinned: repo packageManager is bun@1.4.0, an unpublished version (GH+npm 404) that makes bare setup-bun fail here; match .github/ci-bun-version.json (#15071)
- uses: actions/setup-java@v5
with:
distribution: temurin
java-version: "21"
- uses: android-actions/setup-android@v4
- name: Install workspace
run: bun install
- name: Build WebView-debuggable APK
run: bun run --cwd packages/app build:android
- name: Enable KVM
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
| sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules && sudo udevadm trigger --name-match=kvm
- name: Onboarding→home + chat turn on emulator (host agent)
uses: reactivecircus/android-emulator-runner@v2
with:
api-level: "34"
arch: x86_64
ram-size: 6144M
emulator-options: -no-window -no-snapshot -no-boot-anim -gpu swiftshader_indirect -qemu -cpu qemu64,+avx,+avx2,+f16c,+fma
script: |
set -euo pipefail
adb root || true
adb shell setenforce 0 || true
bun run --cwd packages/app build # ensure web dist
mkdir -p packages/app/test-results/android-onboarding-to-home
ELIZA_API_PORT=31337 \
ELIZA_PAIRING_DISABLED=1 \
node packages/app-core/scripts/run-node-tsx.mjs \
packages/app-core/scripts/serve-real-local-agent.ts \
> packages/app/test-results/android-onboarding-to-home/host-agent.log 2>&1 &
HOST_AGENT_PID=$!
trap 'kill "$HOST_AGENT_PID" 2>/dev/null || true' EXIT
for i in $(seq 1 90); do
if curl -fsS \
-H 'X-ElizaOS-Client-Id: android-pr-device-smoke' \
http://127.0.0.1:31337/api/health >/tmp/android-host-agent-health.json; then
cat /tmp/android-host-agent-health.json
break
fi
if ! kill -0 "$HOST_AGENT_PID" 2>/dev/null; then
echo "Host agent exited before becoming healthy"
cat packages/app/test-results/android-onboarding-to-home/host-agent.log
exit 1
fi
sleep 2
done
curl -fsS \
-H 'X-ElizaOS-Client-Id: android-pr-device-smoke' \
http://127.0.0.1:31337/api/health >/dev/null
# Onboarding→home + first chat turn, asserted against the host agent.
ELIZA_ANDROID_BACKEND=host \
ELIZA_ANDROID_REQUIRE_AGENT=1 \
bun run --cwd packages/app test:e2e:android:onboarding
# Native plugin x WebView smoke: proves Capacitor JS calls cross
# into Android Kotlin rather than the desktop Chromium bridge shim.
# Signal-only (|| true) on the PR lane until it's proven green on the
# x86_64 API-34 emulator this job uses — the hard assertion runs in
# the workflow_dispatch `android-e2e` job (validated on a real arm64
# device). Do not make it a hard PR gate on unproven-on-emulator
# evidence; flip to a hard gate once an emulator-green run is attached.
ELIZA_ANDROID_BACKEND=host \
ELIZA_ANDROID_REQUIRE_AGENT=1 \
bun run --cwd packages/app test:e2e:android:native-plugin-view || true
# Real OS-level touch swipe on the chat-sheet grabber (#9943): proves
# the home→launcher pager gesture fires with real Android touch input
# (pointerMouseCount===0), not desktop mouse emulation. Signal-only on
# the PR lane for the same reason as above — flip to a hard gate once
# emulator-green evidence is attached.
ELIZA_ANDROID_BACKEND=host \
ELIZA_ANDROID_REQUIRE_AGENT=1 \
ELIZA_ANDROID_CLEAR_APP_DATA=1 \
bun run --cwd packages/app test:e2e:android:touch-gesture || true
# View-runtime telemetry soak (#10196): activates every registered
# view through the real eliza:navigate:view channel and asserts
# bounded render/heap/cache behavior. Signal-only on the PR lane.
ELIZA_ANDROID_BACKEND=host \
ELIZA_ANDROID_REQUIRE_AGENT=1 \
bun run --cwd packages/app test:e2e:android:view-runtime-soak || true
- uses: actions/upload-artifact@v7
if: always()
with:
name: android-pr-device-smoke-artifacts
path: |
packages/app/test-results/**
packages/app/playwright-report/**
if-no-files-found: ignore
# On-device instrumented (androidTest) gate for the native Capacitor plugins
# (issue #9967 "wire into CI" + issue #9943 "on-device never gates a PR"). Until
# this lane, the native-plugin Kotlin ran on no test, on no device in CI — the
# only coverage was a mocked Capacitor bridge in desktop Chromium. This builds
# every native plugin's androidTest, boots a KVM emulator, and runs the real
# on-device reads (RoleManager / AudioManager / WifiManager / TelecomManager /
# CameraManager / ContactsProvider / content://sms / UsageStatsManager /
# FusedLocationProvider) via `connectedDebugAndroidTest` — gradle fails the job
# on any test failure. Label-gated (`ci:device`) or dispatch, like the smokes
# above, because booting an emulator is heavy.
native-plugin-androidtest:
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' &&
contains(github.event.pull_request.labels.*.name, 'ci:device'))
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 60
env:
ELIZA_BUN_RISCV64_OPTIONAL: "1"
ELIZA_ANDROID_SKIP_FORK_LLAMA_LIB: "1"
steps:
- uses: actions/checkout@v4
with:
submodules: recursive
- uses: oven-sh/setup-bun@v2
with:
bun-version: "1.3.14" # pinned: repo packageManager is bun@1.4.0, an unpublished version (GH+npm 404) that makes bare setup-bun fail here; match .github/ci-bun-version.json (#15071)
- uses: actions/setup-java@v5
with:
distribution: temurin
java-version: "21"
- uses: android-actions/setup-android@v4
- name: Install workspace
run: bun install
# Sync the Capacitor Android project so capacitor.settings.gradle points at
# this install's @capacitor/android and every plugin module is wired.
- name: Prepare Android platform
run: bun run --cwd packages/app build:android
- name: Enable KVM
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
| sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules && sudo udevadm trigger --name-match=kvm
- name: Native-plugin androidTest on emulator
uses: reactivecircus/android-emulator-runner@v2
with:
api-level: "34"
arch: x86_64
ram-size: 4096M
emulator-options: -no-window -no-snapshot -no-boot-anim -gpu swiftshader_indirect
script: |
set -euo pipefail
adb wait-for-device
ANDROID_DIR="$GITHUB_WORKSPACE/packages/app-core/platforms/android"
GRADLEW="$ANDROID_DIR/gradlew"
chmod +x "$GRADLEW"
# Device-state orchestration (survives the test-APK install):
# - messages: inject a marker SMS so the SMS read-back asserts positively
# (it Assume-skips without the marker, so this is what makes it green).
# - location: high-accuracy mode (the fused fetch still Assume-skips on a
# headless GNSS HAL; the priority-mapping assertion runs regardless).
adb emu sms send 15558675309 "probe Eliza-9967-SMS-roundtrip ci" || true
adb shell settings put secure location_mode 3 || true
# The gate: build + install + run every native-plugin androidTest on the
# emulator in one invocation. gradle fails the job on any test failure.
# `:app:connectedDebugAndroidTest` adds the app-level assistant/IME/
# assist-surface instrumented tests (#13581) — the RETAIL-path
# ElizaAssistantSurfaceInstrumentedTest runs on the plain debug APK
# (not assumeSystemEliza-gated), so this app run is not vacuously
# green off-AOSP.
"$GRADLEW" -p "$ANDROID_DIR" --no-daemon \
:app:connectedDebugAndroidTest \
:elizaos-capacitor-system:connectedDebugAndroidTest \
:elizaos-capacitor-wifi:connectedDebugAndroidTest \
:elizaos-capacitor-phone:connectedDebugAndroidTest \
:elizaos-capacitor-camera:connectedDebugAndroidTest \
:elizaos-capacitor-contacts:connectedDebugAndroidTest \
:elizaos-capacitor-messages:connectedDebugAndroidTest \
:elizaos-capacitor-mobile-signals:connectedDebugAndroidTest \
:elizaos-capacitor-location:connectedDebugAndroidTest
# mobile-signals PACKAGE_USAGE_STATS is special-access (no runtime dialog)
# and the grant does NOT survive the gate's reinstall, so its usage reads
# Assume-skip above. For a positive on-device read: re-install the test APK
# the gate built (the gate uninstalls it after running), grant
# GET_USAGE_STATS, and re-run the usage tests via am instrument. Require a
# clean `OK (N tests)` completion and fail the job on any error/failure.
MS_APK="$GITHUB_WORKSPACE/plugins/plugin-native-mobile-signals/android/build/outputs/apk/androidTest/debug/elizaos-capacitor-mobile-signals-debug-androidTest.apk"
adb install -r -t "$MS_APK"
adb shell appops set ai.eliza.plugins.mobilesignals.test android:get_usage_stats allow
MS_OUT="$(adb shell am instrument -w -r \
-e class ai.eliza.plugins.mobilesignals.UsageStatsReaderInstrumentedTest \
ai.eliza.plugins.mobilesignals.test/androidx.test.runner.AndroidJUnitRunner 2>&1)"
printf '%s\n' "$MS_OUT"
if printf '%s\n' "$MS_OUT" | grep -qE 'FAILURES!!!|INSTRUMENTATION_FAILED|INSTRUMENTATION_RESULT: shortMsg'; then
echo "mobile-signals UsageStats assertions failed under granted PACKAGE_USAGE_STATS"
exit 1
fi
printf '%s\n' "$MS_OUT" | grep -qE 'OK \([0-9]+ test' || {
echo "mobile-signals UsageStats run did not complete cleanly"
exit 1
}
# Assistant-role / voice-IME / assist-key adb verification lane
# (#13581). The gradle gate above uninstalled its test APKs, so
# install the app APK the Capacitor build produced, then drive the
# RUNTIME surface: re-apply the role + IME that `adb install -r`
# clears, assert the secure settings, fire `cmd voiceinteraction
# show` / `KEYCODE_ASSIST` / the IME deep-link, and assert they land
# in MainActivity. `--require-device` makes a missing device fatal
# (it is present here, so this only guards against a silent skip).
# `--require-engine` is intentionally NOT passed: the emulator carries
# no on-device ASR engine, so the IME ASR round-trip legitimately
# resolves to the designed ENGINE_OFF state — the lane asserts that
# state rather than requiring a transcript. All other assertions
# (registration, role, IME selection, deep-link landing) still gate
# the job.
APP_APK="$GITHUB_WORKSPACE/packages/app-core/platforms/android/app/build/outputs/apk/debug/app-debug.apk"
if [ -f "$APP_APK" ]; then
adb install -r -t "$APP_APK"
else
echo "app-debug.apk not found at $APP_APK — build:android should have produced it"
exit 1
fi
ASSISTANT_ARTIFACT_DIR="$GITHUB_WORKSPACE/packages/app/test-results/android-assistant-verify"
mkdir -p "$ASSISTANT_ARTIFACT_DIR"
VERIFY_STATUS=0
node "$GITHUB_WORKSPACE/packages/app/scripts/android-assistant-verify.mjs" \
--require-device --json \
| tee "$ASSISTANT_ARTIFACT_DIR/verdict.json" || VERIFY_STATUS=$?
adb shell settings get secure voice_interaction_service \
> "$ASSISTANT_ARTIFACT_DIR/voice_interaction_service.txt" || true
adb shell settings get secure default_input_method \
> "$ASSISTANT_ARTIFACT_DIR/default_input_method.txt" || true
adb shell ime list -s \
> "$ASSISTANT_ARTIFACT_DIR/enabled_imes.txt" || true
adb shell cmd role holders android.app.role.ASSISTANT \
> "$ASSISTANT_ARTIFACT_DIR/assistant_role_holders.txt" || true
adb shell dumpsys activity activities \
> "$ASSISTANT_ARTIFACT_DIR/dumpsys-activity.txt" || true
adb logcat -d -v brief \
> "$ASSISTANT_ARTIFACT_DIR/logcat.txt" || true
exit "$VERIFY_STATUS"
- uses: actions/upload-artifact@v7
if: always()
with:
name: native-plugin-androidtest-reports
path: |
plugins/plugin-native-*/android/build/reports/androidTests/**
plugins/plugin-native-*/android/build/outputs/androidTest-results/**
packages/app/test-results/android-assistant-verify/**
if-no-files-found: ignore
+396
View File
@@ -0,0 +1,396 @@
name: Android Build & Publish
on:
workflow_call:
inputs:
track:
description: "Play Store track"
required: false
type: string
default: internal
version_name:
description: "Version name (e.g. 2.0.0-beta.0)"
required: true
type: string
secrets:
# ANDROID_KEYSTORE_* are downgraded to `required: false` so the
# reusable-workflow secret contract does not block startup when
# called from release-all.yml / release-orchestrator.yml with
# `secrets: inherit`. The job body's signing step (gated on
# ANDROID_KEYSTORE_BASE64 being non-empty) skips with a clear
# warning when the secrets are absent, instead of aborting the
# whole release pipeline at workflow startup with no logs. Same
# pattern as publish-apt-repo.yml (PR #7976) and update-homebrew.yml.
ANDROID_KEYSTORE_BASE64:
required: false
ANDROID_KEYSTORE_PASSWORD:
required: false
ANDROID_KEY_ALIAS:
required: false
ANDROID_KEY_PASSWORD:
required: false
PLAY_STORE_SERVICE_ACCOUNT_JSON:
required: false
workflow_dispatch:
inputs:
track:
description: "Play Store track"
type: choice
options:
- internal
- beta
- production
default: internal
version_name:
description: "Version name (e.g. 2.0.0-beta.0)"
required: false
type: string
concurrency:
group: android-release-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: write
env:
JAVA_VERSION: "21"
NODE_VERSION: "24.15.0"
jobs:
build-aab:
name: Build Signed AAB
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
outputs:
version_name: ${{ steps.version.outputs.name }}
version_code: ${{ steps.version.outputs.code }}
play_store_ready: ${{ steps.play_store_credentials.outputs.ready }}
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Java
uses: actions/setup-java@ad2b38190b15e4d6bdf0c97fb4fca8412226d287
with:
distribution: "temurin"
java-version: ${{ env.JAVA_VERSION }}
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: "canary"
- name: Determine version
id: version
run: |
if [[ -n "${{ inputs.version_name }}" ]]; then
VERSION_NAME="${{ inputs.version_name }}"
else
VERSION_NAME=$(node -p "require('./package.json').version")
fi
# Generate a monotonic Android version code from semver.
# 2.0.0-alpha.82 -> 200001082, 2.0.0-beta.0 -> 200003000,
# 2.0.0 -> 200009000, 2.1.3 -> 201039000.
VERSION_CODE=$(node - "$VERSION_NAME" <<'NODE'
const version = process.argv[2];
const match = version.match(/^(\d+)\.(\d+)\.(\d+)(?:-([0-9A-Za-z-]+)(?:\.(\d+))?)?$/);
if (!match) {
console.error(`Invalid semver version: ${version}`);
process.exit(1);
}
const [, majorRaw, minorRaw, patchRaw, prerelease, prereleaseRaw = "0"] =
match;
const major = Number(majorRaw);
const minor = Number(minorRaw);
const patch = Number(patchRaw);
const prereleaseNumber = Number(prereleaseRaw);
if (
!Number.isInteger(major) ||
!Number.isInteger(minor) ||
!Number.isInteger(patch) ||
!Number.isInteger(prereleaseNumber) ||
prereleaseNumber < 0 ||
prereleaseNumber > 999
) {
console.error(`Invalid Android version component in ${version}`);
process.exit(1);
}
const channelOffsets = {
alpha: 1000,
beta: 3000,
rc: 5000,
nightly: 7000,
};
const channelOffset = prerelease
? (channelOffsets[prerelease] ?? 8000)
: 9000;
const code =
major * 100000000 +
minor * 1000000 +
patch * 10000 +
channelOffset +
prereleaseNumber;
if (code > 2100000000) {
console.error(`Android versionCode ${code} exceeds Play Store limit`);
process.exit(1);
}
console.log(code);
NODE
)
echo "name=$VERSION_NAME" >> "$GITHUB_OUTPUT"
echo "code=$VERSION_CODE" >> "$GITHUB_OUTPUT"
echo "Version: $VERSION_NAME (code: $VERSION_CODE)"
- name: Check Play Store credentials
id: play_store_credentials
env:
PLAY_STORE_SERVICE_ACCOUNT_JSON: ${{ secrets.PLAY_STORE_SERVICE_ACCOUNT_JSON }}
run: |
if [[ -n "$PLAY_STORE_SERVICE_ACCOUNT_JSON" ]]; then
echo "ready=true" >> "$GITHUB_OUTPUT"
else
echo "ready=false" >> "$GITHUB_OUTPUT"
echo "PLAY_STORE_SERVICE_ACCOUNT_JSON is not configured; signed APK/AAB artifacts will still attach to GitHub Release, but Play Store upload will be skipped."
fi
- name: Install dependencies
run: bun install
- name: Build workspace web runtime packages
run: |
bun run --cwd packages/core build
bun run --cwd packages/shared build
bun run --cwd packages/vault build
bun run --cwd packages/cloud/sdk build
bun run --cwd packages/app-core build:dist
- name: Prepare Android platform
working-directory: packages/app
run: bun run build:android:cloud
- name: Decode keystore
run: |
echo "${{ secrets.ANDROID_KEYSTORE_BASE64 }}" | base64 -d > /tmp/elizaos-upload.jks
env:
ANDROID_KEYSTORE_BASE64: ${{ secrets.ANDROID_KEYSTORE_BASE64 }}
- name: Run Android signing preflight
working-directory: packages/app
env:
ELIZAOS_KEYSTORE_PATH: /tmp/elizaos-upload.jks
ELIZAOS_KEYSTORE_PASSWORD: ${{ secrets.ANDROID_KEYSTORE_PASSWORD }}
ELIZAOS_KEY_ALIAS: ${{ secrets.ANDROID_KEY_ALIAS }}
ELIZAOS_KEY_PASSWORD: ${{ secrets.ANDROID_KEY_PASSWORD }}
run: |
bun run preflight:android:sideload
missing=()
[[ ! -s "$ELIZAOS_KEYSTORE_PATH" ]] && missing+=("ANDROID_KEYSTORE_BASE64")
[[ -z "$ELIZAOS_KEYSTORE_PASSWORD" ]] && missing+=("ANDROID_KEYSTORE_PASSWORD")
[[ -z "$ELIZAOS_KEY_ALIAS" ]] && missing+=("ANDROID_KEY_ALIAS")
[[ -z "$ELIZAOS_KEY_PASSWORD" ]] && missing+=("ANDROID_KEY_PASSWORD")
if [[ ${#missing[@]} -gt 0 ]]; then
echo "::error::Missing Android release signing inputs: ${missing[*]}"
exit 1
fi
- name: Run Play Store preflight
if: steps.play_store_credentials.outputs.ready == 'true'
working-directory: packages/app
env:
ELIZAOS_KEYSTORE_PATH: /tmp/elizaos-upload.jks
ELIZAOS_KEYSTORE_PASSWORD: ${{ secrets.ANDROID_KEYSTORE_PASSWORD }}
ELIZAOS_KEY_ALIAS: ${{ secrets.ANDROID_KEY_ALIAS }}
ELIZAOS_KEY_PASSWORD: ${{ secrets.ANDROID_KEY_PASSWORD }}
PLAY_STORE_SERVICE_ACCOUNT_JSON: ${{ secrets.PLAY_STORE_SERVICE_ACCOUNT_JSON }}
run: bun run preflight:android:store
- name: Build signed AAB
working-directory: packages/app-core/platforms/android
env:
ELIZAOS_VERSION_NAME: ${{ steps.version.outputs.name }}
ELIZAOS_VERSION_CODE: ${{ steps.version.outputs.code }}
ELIZAOS_KEYSTORE_PATH: /tmp/elizaos-upload.jks
ELIZAOS_KEYSTORE_PASSWORD: ${{ secrets.ANDROID_KEYSTORE_PASSWORD }}
ELIZAOS_KEY_ALIAS: ${{ secrets.ANDROID_KEY_ALIAS }}
ELIZAOS_KEY_PASSWORD: ${{ secrets.ANDROID_KEY_PASSWORD }}
run: |
chmod +x gradlew
./gradlew -PelizaCloudBuild=true -PelizaStripAgentAssets=true bundleRelease
- name: Build signed APK
working-directory: packages/app-core/platforms/android
env:
ELIZAOS_VERSION_NAME: ${{ steps.version.outputs.name }}
ELIZAOS_VERSION_CODE: ${{ steps.version.outputs.code }}
ELIZAOS_KEYSTORE_PATH: /tmp/elizaos-upload.jks
ELIZAOS_KEYSTORE_PASSWORD: ${{ secrets.ANDROID_KEYSTORE_PASSWORD }}
ELIZAOS_KEY_ALIAS: ${{ secrets.ANDROID_KEY_ALIAS }}
ELIZAOS_KEY_PASSWORD: ${{ secrets.ANDROID_KEY_PASSWORD }}
run: |
chmod +x gradlew
./gradlew -PelizaCloudBuild=true -PelizaStripAgentAssets=true assembleRelease
- name: Verify Android release artifacts
run: |
AAB_PATH="packages/app-core/platforms/android/app/build/outputs/bundle/release/app-release.aab"
APK_PATH="packages/app-core/platforms/android/app/build/outputs/apk/release/app-release.apk"
if [[ ! -f "$AAB_PATH" ]]; then
echo "ERROR: AAB not found at $AAB_PATH"
ls -la packages/app-core/platforms/android/app/build/outputs/bundle/release/ 2>/dev/null || echo "Release dir not found"
exit 1
fi
if [[ ! -f "$APK_PATH" ]]; then
echo "ERROR: APK not found at $APK_PATH"
ls -la packages/app-core/platforms/android/app/build/outputs/apk/release/ 2>/dev/null || echo "Release dir not found"
exit 1
fi
echo "AAB size: $(du -h "$AAB_PATH" | cut -f1)"
echo "APK size: $(du -h "$APK_PATH" | cut -f1)"
file "$AAB_PATH"
file "$APK_PATH"
sha256sum "$AAB_PATH" "$APK_PATH" > packages/app-core/platforms/android/app/build/outputs/release-SHA256SUMS.txt
cat packages/app-core/platforms/android/app/build/outputs/release-SHA256SUMS.txt
- name: Stage sideload APK with canonical name
run: |
VERSION="${{ steps.version.outputs.name }}"
APK_SRC="packages/app-core/platforms/android/app/build/outputs/apk/release/app-release.apk"
SIDELOAD_APK="elizaos-android-${VERSION}-release.apk"
cp "$APK_SRC" "$SIDELOAD_APK"
sha256sum "$SIDELOAD_APK" > "${SIDELOAD_APK}.sha256"
echo "Sideload APK: $SIDELOAD_APK"
echo "SHA256: $(cat "${SIDELOAD_APK}.sha256")"
- name: Upload Android release artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: eliza-android-release
path: |
packages/app-core/platforms/android/app/build/outputs/bundle/release/app-release.aab
packages/app-core/platforms/android/app/build/outputs/apk/release/app-release.apk
packages/app-core/platforms/android/app/build/outputs/release-SHA256SUMS.txt
elizaos-android-${{ steps.version.outputs.name }}-release.apk
elizaos-android-${{ steps.version.outputs.name }}-release.apk.sha256
retention-days: 90
- name: Attach Android artifacts to GitHub Release
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
VERSION="${{ steps.version.outputs.name }}"
TAG="v${VERSION}"
SIDELOAD_APK="elizaos-android-${VERSION}-release.apk"
if ! gh release view "$TAG" >/dev/null 2>&1; then
echo "::error::No GitHub release found for $TAG; cannot attach AAB."
exit 1
fi
cp packages/app-core/platforms/android/app/build/outputs/bundle/release/app-release.aab \
"Eliza-${VERSION}.aab"
cp packages/app-core/platforms/android/app/build/outputs/apk/release/app-release.apk \
"Eliza-${VERSION}.apk"
cp packages/app-core/platforms/android/app/build/outputs/release-SHA256SUMS.txt \
"Eliza-${VERSION}.android.SHA256SUMS.txt"
gh release upload "$TAG" \
"Eliza-${VERSION}.aab" \
"Eliza-${VERSION}.apk" \
"Eliza-${VERSION}.android.SHA256SUMS.txt" \
"${SIDELOAD_APK}" \
"${SIDELOAD_APK}.sha256" \
--clobber
- name: Clean up keystore
if: always()
run: rm -f /tmp/elizaos-upload.jks
publish-play-store:
name: Publish to Play Store
needs: build-aab
if: needs.build-aab.outputs.play_store_ready == 'true'
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Download AAB
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c
with:
name: eliza-android-release
path: aab/
- name: Setup Ruby
uses: ruby/setup-ruby@d45b1a4e94b71acab930e56e79c6aa188764e7f9
with:
ruby-version: "4.0.3"
bundler-cache: true
working-directory: packages/app-core/platforms/android
- name: Decode Play Store service account key
run: |
echo "${{ secrets.PLAY_STORE_SERVICE_ACCOUNT_JSON }}" | base64 -d > /tmp/play-store-key.json
- name: Determine track
id: track
run: |
if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
echo "track=${{ inputs.track }}" >> "$GITHUB_OUTPUT"
elif echo "${{ needs.build-aab.outputs.version_name }}" | grep -qE '(beta|rc)'; then
echo "track=internal" >> "$GITHUB_OUTPUT"
else
echo "track=production" >> "$GITHUB_OUTPUT"
fi
- name: Upload to Play Store
working-directory: packages/app-core/platforms/android
env:
PLAY_STORE_JSON_KEY: /tmp/play-store-key.json
run: |
bundle exec fastlane supply \
--aab "$GITHUB_WORKSPACE/aab/app-release.aab" \
--track "${{ steps.track.outputs.track }}" \
--json_key "$PLAY_STORE_JSON_KEY" \
--package_name "app.eliza"
- name: Clean up credentials
if: always()
run: rm -f /tmp/play-store-key.json
summary:
name: Release Summary
needs: [build-aab, publish-play-store]
if: always()
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- name: Summary
run: |
{
echo "## Android Release Summary"
echo ""
echo "**Version:** ${{ needs.build-aab.outputs.version_name }} (code: ${{ needs.build-aab.outputs.version_code }})"
echo ""
echo "| Step | Status |"
echo "|------|--------|"
echo "| Build AAB | ${{ needs.build-aab.result }} |"
echo "| Publish to Play Store | ${{ needs.publish-play-store.result }} |"
} >> "$GITHUB_STEP_SUMMARY"
- name: Require Android release succeeded
run: |
failed=0
if [[ "${{ needs.build-aab.result }}" != "success" ]]; then
echo "::error::Signed AAB build result was '${{ needs.build-aab.result }}', expected success"
failed=1
fi
if [[ "${{ needs.build-aab.outputs.play_store_ready }}" == "true" && "${{ needs.publish-play-store.result }}" != "success" ]]; then
echo "::error::Play Store publish result was '${{ needs.publish-play-store.result }}', expected success"
failed=1
fi
exit "$failed"
+131
View File
@@ -0,0 +1,131 @@
name: App Aesthetic Audit
# Walk EVERY app view (built-in tabs + plugin view bundles) at desktop + mobile
# against the deterministic keyless stub stack, capture rest + hover screenshots,
# run the blank/one-color analyzer + brand-color (no-blue / orange-hover) checks,
# and write per-view verdicts + a contact sheet + report.json.
#
# This is the agent app's equivalent of cloud-frontend's `audit:cloud`. Until
# this workflow existed the audit ran in NO CI lane (honor-system per the PR
# template, #9304). It runs the strict verifier and uploads all artifacts, but
# remains advisory for PR throughput (#14051): the required `ci-ok` gate owns
# merge blocking, while this lane supplies visual evidence for reviewers.
on:
pull_request:
branches: [main, develop]
paths:
- "packages/app/**"
- "packages/ui/**"
- "packages/shared/**"
- "plugins/**"
- ".github/workflows/app-aesthetic-audit.yml"
workflow_dispatch:
concurrency:
group: app-aesthetic-audit-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
BUN_VERSION: "canary"
NODE_VERSION: "24.15.0"
jobs:
aesthetic-audit:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 45
continue-on-error: true
env:
# Keyless: the ui-smoke stub is forced when CI=true (shouldForceStubStack).
ELIZA_UI_SMOKE_FORCE_STUB: "1"
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Install Playwright Chromium
run: .github/scripts/install-playwright-browsers.sh chromium
- name: Run all-views aesthetic audit
# Strict verifier, advisory check (#14051): failures stay visible in
# logs/artifacts without blocking the required ci-ok merge gate.
continue-on-error: true
env:
ELIZA_AUDIT_APP_STRICT: "1"
ELIZA_AUDIT_APP_STRICT_NEEDS_WORK: "1"
run: bun run --cwd packages/app audit:app:capture
- name: Upload aesthetic-audit artifacts
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: app-aesthetic-audit-output
path: packages/app/aesthetic-audit-output
retention-days: 14
if-no-files-found: ignore
ocr-content-audit:
needs: aesthetic-audit
runs-on: macos-15
timeout-minutes: 20
continue-on-error: true
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Download aesthetic-audit artifacts
continue-on-error: true
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c
with:
name: app-aesthetic-audit-output
path: packages/app/aesthetic-audit-output
- name: Ensure OCR audit output directory exists
run: mkdir -p packages/app/aesthetic-audit-output
- name: Run pixel OCR content audit
continue-on-error: true
env:
ELIZA_MVP_OCR_ENGINE: packaged
run: bun run --cwd packages/app audit:ocr
- name: Upload OCR audit artifacts
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: app-ocr-audit-output
path: packages/app/aesthetic-audit-output/ocr-triage.json
retention-days: 14
if-no-files-found: ignore
+445
View File
@@ -0,0 +1,445 @@
name: App Live E2E
# Gated, NON-blocking live lane for the packages/app e2e suite — the home for
# REAL (not stub/mock) coverage of the dimensions the keyless PR lanes can only
# fixture-test.
#
# Why the keyless lanes can't do this: every PR-gating lane (scenario-pr, ci,
# test) forces the deterministic stub API (CI=true => FORCE_STUB_STACK, see the
# tested shouldForceStubStack helper) and blanks all provider/cloud keys, because
# secrets cannot be exposed to fork PRs and real model/cloud turns cost money and
# add network flake. So real coverage MUST live in a secret-gated, non-PR lane.
#
# The keystone that makes this possible: ELIZA_UI_SMOKE_LIVE_STACK=1 overrides the
# CI stub-force, so the live stack boots the real @elizaos/app-core runtime and
# the live-only specs (test.skip'd in keyless CI) un-skip.
#
# Runs nightly + on-demand; never on pull_request.
on:
schedule:
# 07:00 UTC daily — after the nightly build lanes settle.
- cron: "0 7 * * *"
workflow_dispatch:
inputs:
run_android_local_chat:
description: "Also run the on-device Android local-chat smoke (boots an emulator + builds/installs the APK + stages a GGUF)."
type: boolean
default: false
concurrency:
group: app-live-e2e-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
CI: "true"
# The keystone: opt the live stack out of the CI-based stub force so the real
# app-core runtime boots. Without this, CI=true re-forces the stub even with a
# provider key present and every "live" test self-skips.
ELIZA_UI_SMOKE_LIVE_STACK: "1"
BUN_VERSION: "canary"
NODE_VERSION: "24.15.0"
NODE_NO_WARNINGS: "1"
CEREBRAS_API_KEY: ${{ secrets.CEREBRAS_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_GENERATIVE_AI_API_KEY }}
XAI_API_KEY: ${{ secrets.XAI_API_KEY }}
jobs:
# Real LOCAL chat turn through the live stack: a real app-core runtime + a real
# provider model, asserting an exact model-produced marker (APP_LIVE_AGENT_OK).
# UI-driven counterpart to dev-smoke; proves chat send/stream reaches a real
# agent from the UI.
app-live-chat:
name: app live chat (real local runtime)
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 45
env:
PGLITE_WASM_MODE: node
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Free disk space for browser smoke
run: |
set -euxo pipefail
df -h /
sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
docker system prune -af || true
df -h /
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Install Playwright Chromium
run: bunx playwright install --with-deps chromium
- name: Generate shared i18n assets
run: bun run --cwd packages/shared build:i18n
# Vite loads package exports before renderer aliases apply; fresh CI
# checkouts need these dist bundles before packages/app can resolve
# @elizaos/core and @elizaos/shared subpaths.
- name: Build @elizaos/core + @elizaos/shared browser bundles
run: bunx turbo run build --filter=@elizaos/core --filter=@elizaos/shared
# Prebuild the renderer dist so the live stack reuses a fresh bundle
# instead of a cold in-harness vite build (viteRendererBuildNeeded check).
- name: Build app renderer bundle
run: bun run --cwd packages/app build:web
- name: Run live agent chat (real model turn)
run: bun run --cwd packages/app test:e2e test/ui-smoke/live-agent-chat.spec.ts --project=chromium -g "live agent"
# Deep settings/vault round-trips that the keyless stub structurally
# cannot serve (write→reload→read-back against the real on-disk vault +
# runtime). LIVE_STACK=1 is set job-wide, so the test.skip(!LIVE_STACK)
# guards in these specs open. The two *-interactions specs also run keyless
# in scenario-pr.yml (skipping their deep blocks); here they run them.
- name: Run settings + vault live-stack deep e2e
run: bun run --cwd packages/app test:e2e test/ui-smoke/vault-routing.spec.ts test/ui-smoke/wallet-keys.spec.ts test/ui-smoke/provider-config.spec.ts test/ui-smoke/vault-modal-interactions.spec.ts test/ui-smoke/settings-sections-interactions.spec.ts --project=chromium
- name: Upload live-chat artifacts
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: app-live-e2e-chat
path: |
packages/app/playwright-report/
packages/app/test-results/
if-no-files-found: ignore
retention-days: 7
# Full continuous walkthrough against the REAL backend agent + model: cold
# launch → in-chat onboarding (greeting → runtime CHOICE → provider → one real
# POST /api/first-run) → all 8 tutorial frames → help/settings/wallet → a real
# chat round-trip (trajectory captured) → view switching → settings edit. Runs
# WITHOUT --skip-review so the AI vision reviewer scores every NN-step.png and
# writes WALKTHROUGH_VERDICTS.md (the PR mock lane is keyless and can't).
walkthrough-live:
name: full walkthrough (real backend + model + vision verdicts)
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 60
env:
PGLITE_WASM_MODE: node
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Free disk space for browser smoke
run: |
set -euxo pipefail
sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
docker system prune -af || true
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Install Playwright Chromium
run: bunx playwright install --with-deps chromium
- name: Install ffmpeg (recording stitch)
run: sudo apt-get update && sudo apt-get install -y ffmpeg
- name: Generate shared i18n assets
run: bun run --cwd packages/shared build:i18n
# Vite resolves linked workspace package exports while loading the app
# config, before renderer aliases can redirect to source files.
- name: Build @elizaos/core + @elizaos/shared browser bundles
run: bunx turbo run build --filter=@elizaos/core --filter=@elizaos/shared
# Prebuild the renderer dist so the live stack reuses a fresh bundle
# instead of a cold in-harness vite build.
- name: Build app renderer bundle
run: bun run --cwd packages/app build:web
- name: Drive + record + vision-review the full walkthrough (live)
run: bun run --cwd packages/app test:e2e:walkthrough:live
- name: Upload walkthrough screenshots + recording + verdicts
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: app-live-e2e-walkthrough
path: |
reports/walkthrough/
e2e-recordings/app/walkthrough/
packages/app/test/ui-smoke/walkthrough/WALKTHROUGH_VERDICTS.md
if-no-files-found: ignore
retention-days: 7
# Real CLOUD login + provisioning + chat through the app UI against real Eliza
# Cloud. cloud-live.spec.ts mocks nothing; ELIZA_UI_SMOKE_CLOUD_LIVE=1 leaves
# first-run open so the spec drives cloud onboarding, and ELIZAOS_CLOUD_API_KEY
# authenticates. The secret-gated job fails before setup when neither accepted
# repository secret is configured; the spec remains safely skippable outside
# this workflow so keyless PR lanes cannot spend Cloud credits.
cloud-live:
name: cloud login + provision + chat (real Eliza Cloud)
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 45
env:
ELIZA_UI_SMOKE_CLOUD_LIVE: "1"
# Prefer the variable named for the runtime contract, but retain the
# repository's established ELIZACLOUD_API_KEY as the canonical fallback.
ELIZAOS_CLOUD_API_KEY: ${{ secrets.ELIZAOS_CLOUD_API_KEY || secrets.ELIZACLOUD_API_KEY }}
PGLITE_WASM_MODE: node
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Require real Cloud credential
shell: bash
run: |
set -euo pipefail
cloud_key_without_whitespace="${ELIZAOS_CLOUD_API_KEY:-}"
cloud_key_without_whitespace="${cloud_key_without_whitespace//[[:space:]]/}"
if [[ -z "$cloud_key_without_whitespace" ]]; then
echo "::error::App Live E2E requires ELIZAOS_CLOUD_API_KEY or ELIZACLOUD_API_KEY; refusing a green-by-skip Cloud job."
exit 1
fi
unset cloud_key_without_whitespace
- name: Free disk space for browser smoke
run: |
set -euxo pipefail
sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
docker system prune -af || true
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Install Playwright Chromium
run: bunx playwright install --with-deps chromium
- name: Generate shared i18n assets
run: bun run --cwd packages/shared build:i18n
# Cloud-live still builds the same renderer bundle, so it needs the same
# workspace dists before Vite can load packages/app/vite.config.ts.
- name: Build @elizaos/core + @elizaos/shared browser bundles
run: bunx turbo run build --filter=@elizaos/core --filter=@elizaos/shared
- name: Build app renderer bundle
run: bun run --cwd packages/app build:web
- name: Run real cloud login + provision + chat
run: bun run --cwd packages/app test:e2e test/ui-smoke/cloud-live.spec.ts --project=chromium
- name: Upload cloud-live artifacts
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: app-live-e2e-cloud
path: |
packages/app/playwright-report/
packages/app/test-results/
if-no-files-found: ignore
retention-days: 7
# Real on-device LOCAL provisioning + chat on Android: boots an emulator, builds
# + installs the debuggable APK, stages the smoke GGUF, starts the native
# ElizaAgentService local runtime, and asserts a real streamed on-device model
# reply (test:sim:local-chat:android:live). The ONLY lane that exercises Android
# local provisioning for real. Opt-in (workflow_dispatch input) because it is
# heavy. android-emulator-runner is the de-facto standard emulator action.
android-local-chat:
name: android local chat (on-device GGUF)
if: ${{ github.event_name == 'workflow_dispatch' && inputs.run_android_local_chat }}
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 90
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Enable KVM for the Android emulator
run: |
echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
| sudo tee /etc/udev/rules.d/99-kvm4all.rules
sudo udevadm control --reload-rules
sudo udevadm trigger --name-match=kvm
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup JDK 21
uses: actions/setup-java@ad2b38190b15e4d6bdf0c97fb4fca8412226d287
with:
distribution: temurin
java-version: "21"
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
# android-emulator-runner provides the booted AVD + adb; the script builds
# and installs the APK, then runs the live on-device local-chat smoke, which
# stages the GGUF and asserts a real model reply.
- name: Build, install, and run Android on-device local-chat smoke
uses: reactivecircus/android-emulator-runner@62dbb605bba737720e10b196cb4220d374026a6d
with:
api-level: 34
arch: x86_64
force-avd-creation: false
emulator-options: -no-snapshot-save -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim
disable-animations: true
script: |
bun run --cwd packages/app build:android
bun run --cwd packages/app install:android:adb
bun run --cwd packages/app test:sim:local-chat:android:live
# Desktop (Electrobun) packaged e2e. The build + agent boot run on a standard
# runner and catch real packaged-app bugs (e.g. a missing JSON import attribute
# that breaks the wallet plugin on boot). The renderer-driving assertions need a
# real GPU display (the app uses WebGPU); on GPU-less runners the build + launch
# still validate, so the renderer step is best-effort.
desktop-packaged:
name: desktop packaged (build + boot + renderer)
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 60
env:
PGLITE_WASM_MODE: node
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Install WebKitGTK + xvfb for the Electrobun webview
# libayatana-appindicator3-1: libNativeWrapper.so (tray support) links
# against it; without it every packaged spec dies at launch with
# ERR_DLOPEN_FAILED before the test bridge comes up (the lane was red
# for 5+ days with "Packaged app exited before the desktop test bridge
# became ready").
# scrot/imagemagick/x11-apps/ffmpeg: ScreenCaptureManager tries, in
# order, scrot, ImageMagick import, ffmpeg x11grab, then xwd+ffmpeg for
# /main-window/screenshot. ffmpeg also stitches the walkthrough MP4.
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends \
xvfb libwebkit2gtk-4.1-0 libgtk-3-0 mesa-utils libgl1-mesa-dri \
libayatana-appindicator3-1 scrot imagemagick x11-apps ffmpeg
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Install Playwright runner
run: bunx playwright install --with-deps chromium
# Build the packaged desktop app (catches build/packaging breaks). Run from
# the repo root so desktop-build.mjs resolves packages/app.
- name: Build the Electrobun desktop app
run: node packages/app-core/scripts/desktop-build.mjs build
# Launch + drive the packaged app under a virtual display + software GL.
- name: Run packaged desktop e2e (xvfb)
run: xvfb-run -a --server-args="-screen 0 1920x1080x24" bun run --cwd packages/app test:desktop:packaged
- name: Upload desktop-packaged artifacts
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: app-live-e2e-desktop
path: |
packages/app/playwright-report/
packages/app/test-results/
if-no-files-found: ignore
retention-days: 7
# A nightly that is always red is indistinguishable from no lane: without an
# alert, nobody notices the desktop loop has no green unattended cadence and
# regressions accumulate silently (issue #13681). This job turns a red
# scheduled run into an actionable signal by commenting on the tracking issue
# with the per-job results and a link to the run. Scheduled runs only —
# manual workflow_dispatch runs stay quiet to avoid noise.
notify-on-failure:
name: notify on red nightly
needs: [app-live-chat, walkthrough-live, cloud-live, desktop-packaged]
if: ${{ always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure') }}
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
permissions:
issues: write
steps:
- name: Comment red-nightly diagnostic on tracking issue #13681
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
R_CHAT: ${{ needs.app-live-chat.result }}
R_WALK: ${{ needs.walkthrough-live.result }}
R_CLOUD: ${{ needs.cloud-live.result }}
R_DESKTOP: ${{ needs.desktop-packaged.result }}
run: |
set -euo pipefail
{
printf '## 🌙 Nightly App Live E2E failed (scheduled run)\n\n'
printf 'The only scheduled desktop-packaged lane went red. Per-job results:\n\n'
printf -- '- app live chat: `%s`\n' "$R_CHAT"
printf -- '- full walkthrough: `%s`\n' "$R_WALK"
printf -- '- cloud login+provision+chat: `%s`\n' "$R_CLOUD"
printf -- '- desktop packaged (build+boot+renderer): `%s`\n' "$R_DESKTOP"
printf '\nRun log + failure artifacts: %s\n\n' "$RUN_URL"
printf '_Auto-posted by app-live-e2e.yml so a red nightly notifies someone (Done-when, #13681)._\n'
} > "$RUNNER_TEMP/nightly-red.md"
gh issue comment 13681 --repo "$REPO" --body-file "$RUNNER_TEMP/nightly-red.md"
+91
View File
@@ -0,0 +1,91 @@
name: App Real E2E (browser-driven)
# Gated, NON-blocking lane for the browser-driven REAL app e2e suite that lived
# in no workflow until now (it was dark): test/app/*.{real,live}.e2e.test.ts —
# - qa-checklist.real.e2e.test.ts (onboarding + doc upload + chat +
# settings pages + voice + char switch)
# - memory-relationships.real.e2e.test.ts(real /memories + /relationships views)
# - streaming-visible-text.live.e2e.test.ts (real SSE token streaming)
#
# These drive a real renderer (puppeteer/playwright) against a real app-core
# runtime + a real model provider. Each self-skips (describeIf / CAN_RUN) unless
# ELIZA_LIVE_TEST=1 AND a provider is resolvable, so a key-less run is a clean
# no-op rather than a failure. The default vitest config EXCLUDES these files and
# only scans src/, so they need the dedicated vitest.app-real-e2e.config.ts +
# the test:app-real-e2e script (added alongside this workflow).
#
# Runs nightly + on-demand; never on pull_request.
on:
schedule:
# 09:00 UTC daily — after app-live-e2e (07:00) and voice-live-e2e (08:30).
- cron: "0 9 * * *"
workflow_dispatch: {}
concurrency:
group: app-real-e2e-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
CI: "true"
NODE_VERSION: "24"
BUN_VERSION: "canary"
ELIZA_LIVE_TEST: "1"
# Provider keys (the suite picks the first resolvable via selectLiveProvider();
# absent secrets => the suite self-skips, never fails).
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
CEREBRAS_API_KEY: ${{ secrets.CEREBRAS_API_KEY }}
GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_GENERATIVE_AI_API_KEY }}
ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY }}
jobs:
app-real-e2e:
name: Browser-driven real app e2e (nightly)
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 60
continue-on-error: true
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install dependencies
run: bun install --frozen-lockfile
- name: Install browser drivers
run: |
set -euo pipefail
bunx playwright install --with-deps chromium
- name: Run browser-driven real app e2e
run: |
set -euo pipefail
# streaming-visible-text.live.e2e launches a real browser via
# playwright-core's chromium and honors ELIZA_CHROME_PATH, which
# defaults to a macOS Google Chrome path that never exists on ubuntu —
# so without this the live streaming suite self-skips forever. Point it
# at the chromium the "Install browser drivers" step installed, resolved
# from the same playwright-core the test imports.
ELIZA_CHROME_PATH="$(cd packages/app-core && node -e "console.log(require('playwright-core').chromium.executablePath())")"
test -x "$ELIZA_CHROME_PATH"
echo "Using ELIZA_CHROME_PATH=$ELIZA_CHROME_PATH"
export ELIZA_CHROME_PATH
bun run --cwd packages/app-core test:app-real-e2e
+551
View File
@@ -0,0 +1,551 @@
name: Apple Store Build & Publish
on:
workflow_call:
inputs:
platform:
description: "Platform to build"
required: false
type: string
default: both
track:
description: "Release track"
required: false
type: string
default: testflight
version:
description: "Version override (e.g. 2.0.0-beta.0)"
required: true
type: string
secrets:
# APPLE_ID, APPLE_TEAM_ID, APPLE_APP_SPECIFIC_PASSWORD are
# downgraded to `required: false` so this reusable workflow's
# secret contract does not startup_failure the parent
# (release-all.yml / release-orchestrator.yml) when called via
# `secrets: inherit` and the secrets are absent. The body's
# iOS-signing preflight checks for missing iOS secrets and
# exits with a clear ::error::, so the job still fails loudly
# when it can't sign — just inside the job, not at the parent
# workflow's startup. Same pattern as publish-apt-repo.yml
# (PR #7976), android-release.yml, update-homebrew.yml.
APPLE_ID:
required: false
APPLE_TEAM_ID:
required: false
ITC_TEAM_ID:
required: false
APP_STORE_APP_ID:
required: false
MATCH_PASSWORD:
required: false
MATCH_GIT_URL:
required: false
MATCH_GIT_BASIC_AUTHORIZATION:
required: false
APPLE_APP_SPECIFIC_PASSWORD:
required: false
MAS_CSC_LINK:
required: false
MAS_CSC_KEY_PASSWORD:
required: false
MAS_INSTALLER_CERT:
required: false
MAS_INSTALLER_KEY_PASSWORD:
required: false
APP_STORE_API_KEY_ID:
required: false
APP_STORE_API_ISSUER_ID:
required: false
APP_STORE_API_KEY_P8:
required: false
workflow_dispatch:
inputs:
platform:
description: "Platform to build"
type: choice
options:
- both
- ios
- macos
default: both
track:
description: "Release track"
type: choice
options:
- testflight
- app-store
default: testflight
version:
description: "Version override (e.g. 2.0.0-beta.0)"
required: false
type: string
concurrency:
group: apple-release-${{ github.ref }}
cancel-in-progress: false
env:
BUN_VERSION: "canary"
NODE_VERSION: "24.15.0"
LANG: en_US.UTF-8
LC_ALL: en_US.UTF-8
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
prepare:
name: Prepare
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
outputs:
version: ${{ steps.ver.outputs.version }}
build_number: ${{ steps.ver.outputs.build_number }}
build_ios: ${{ steps.targets.outputs.ios }}
build_macos: ${{ steps.targets.outputs.macos }}
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Determine version
id: ver
run: |
if [[ -n "${{ inputs.version }}" ]]; then
VERSION="${{ inputs.version }}"
else
VERSION=$(node -p "require('./package.json').version")
fi
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
echo "build_number=$(date +%s)" >> "$GITHUB_OUTPUT"
echo "Version: $VERSION"
- name: Determine build targets
id: targets
run: |
PLATFORM="${{ inputs.platform || 'both' }}"
if [[ "$PLATFORM" == "both" || "$PLATFORM" == "ios" ]]; then
echo "ios=true" >> "$GITHUB_OUTPUT"
else
echo "ios=false" >> "$GITHUB_OUTPUT"
fi
if [[ "$PLATFORM" == "both" || "$PLATFORM" == "macos" ]]; then
echo "macos=true" >> "$GITHUB_OUTPUT"
else
echo "macos=false" >> "$GITHUB_OUTPUT"
fi
# ── iOS App Store ────────────────────────────────────────────────────
build-ios:
name: Build & Submit iOS
needs: prepare
if: needs.prepare.outputs.build_ios == 'true'
runs-on: macos-15
# Embedding the no-JIT Bun engine + CocoaPods adds build time over a thin
# cloud-only client, so allow more headroom than the bare-IPA budget.
timeout-minutes: 90
env:
# Make this split build (web + cap sync + ios-overlay + fastlane) behave
# like buildIos()'s App Store target, which bundles a working on-device
# agent. These env vars are normally set by configureIosAppStoreBuildDefaults
# / buildWeb("ios"); the workflow must set them itself because it bypasses
# buildIos(). Without them the IPA ships as a cloud-only thin client whose
# "start local agent" path hard-fails ("the JSContext compatibility
# transport is disabled outside iOS development builds").
# - ELIZA_BUILD_VARIANT/RELEASE_AUTHORITY: bake __ELIZA_BUILD_VARIANT__
# ="store" (store CSP + isNativeIosStoreBuild()) and embed the engine.
# - ELIZA_IOS_FULL_BUN_ENGINE=1: force the engine pod into the IPA.
# - VITE_ELIZA_IOS_FULL_BUN_AVAILABLE=1: tell the renderer the engine is
# present (isFullBunRuntimeBuiltIn) so the transport actually uses it.
# - VITE_ELIZA_IOS_RUNTIME_MODE=cloud-hybrid: App Store default mode; the
# user switching to "local" persists to localStorage and wins.
ELIZA_BUILD_VARIANT: store
ELIZA_RELEASE_AUTHORITY: apple-app-store
ELIZA_IOS_FULL_BUN_ENGINE: "1"
VITE_ELIZA_IOS_FULL_BUN_AVAILABLE: "1"
VITE_ELIZA_IOS_RUNTIME_MODE: cloud-hybrid
APPLE_ID: ${{ secrets.APPLE_ID }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
ITC_TEAM_ID: ${{ secrets.ITC_TEAM_ID }}
APP_STORE_APP_ID: ${{ secrets.APP_STORE_APP_ID }}
MATCH_PASSWORD: ${{ secrets.MATCH_PASSWORD }}
MATCH_GIT_URL: ${{ secrets.MATCH_GIT_URL }}
MATCH_GIT_BASIC_AUTHORIZATION: ${{ secrets.MATCH_GIT_BASIC_AUTHORIZATION }}
FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD: ${{ secrets.APPLE_APP_SPECIFIC_PASSWORD }}
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install dependencies
run: bun install --ignore-scripts || bun install --ignore-scripts
- name: Materialize bun npm package binary
run: if [ -f node_modules/bun/install.js ]; then (cd node_modules/bun && node install.js); fi
- name: Set version
env:
RELEASE_VERSION: ${{ needs.prepare.outputs.version }}
run: |
node -e "
const fs = require('fs');
const v = process.env.RELEASE_VERSION;
for (const f of ['package.json', 'packages/app/package.json']) {
const pkg = JSON.parse(fs.readFileSync(f, 'utf8'));
pkg.version = v;
fs.writeFileSync(f, JSON.stringify(pkg, null, 2) + '\n');
}
"
- name: Build web assets
working-directory: packages/app
run: bun run build
- name: Validate iOS secrets
run: |
# Secrets required for signing the IPA. Without these the build
# itself cannot produce a signed artifact.
missing=()
[[ -z "$MATCH_GIT_URL" ]] && missing+=("MATCH_GIT_URL")
[[ -z "$MATCH_PASSWORD" ]] && missing+=("MATCH_PASSWORD")
[[ -z "$MATCH_GIT_BASIC_AUTHORIZATION" ]] && missing+=("MATCH_GIT_BASIC_AUTHORIZATION")
[[ -z "$ITC_TEAM_ID" ]] && missing+=("ITC_TEAM_ID")
if [[ ${#missing[@]} -gt 0 ]]; then
echo "::error::Missing required iOS signing secrets: ${missing[*]}"
echo "Configure these secrets in the repo settings to enable iOS App Store builds."
exit 1
fi
if [[ -z "$APP_STORE_APP_ID" ]]; then
echo "::error::APP_STORE_APP_ID is required for TestFlight/App Store delivery."
exit 1
fi
- name: Prepare CocoaPods trunk repo
run: bash packages/app-core/scripts/prepare-ios-cocoapods.sh
env:
LANG: en_US.UTF-8
- name: Sync Capacitor iOS
working-directory: packages/app
run: bun run cap:sync:ios
env:
LANG: en_US.UTF-8
# Build the on-device agent bundle the embedded Bun engine executes.
# buildIos() does this via buildMobileAgentBundle(); this split workflow
# must run it explicitly so the ios-overlay step below can stage
# packages/agent/dist-mobile-ios into the app's public/agent.
- name: Build iOS agent bundle
run: bun run --cwd packages/agent build:ios-bun
- name: Overlay iOS native files (Fastlane, AppDelegate, entitlements)
run: node packages/app-core/scripts/run-mobile-build.mjs ios-overlay
env:
LANG: en_US.UTF-8
- name: Update Xcode project version
working-directory: packages/app/ios/App/App.xcodeproj
run: |
VERSION="${{ needs.prepare.outputs.version }}"
BUILD="${{ needs.prepare.outputs.build_number }}"
sed -i '' "s/MARKETING_VERSION = .*/MARKETING_VERSION = $VERSION;/" project.pbxproj
sed -i '' "s/CURRENT_PROJECT_VERSION = .*/CURRENT_PROJECT_VERSION = $BUILD;/" project.pbxproj
- name: Install CocoaPods
working-directory: packages/app/ios/App
run: pod install
- name: Setup Ruby
uses: ruby/setup-ruby@d45b1a4e94b71acab930e56e79c6aa188764e7f9
with:
ruby-version: "4.0.3"
bundler-cache: true
working-directory: packages/app/ios
- name: Resolve app identifiers
working-directory: packages/app
run: |
# Read the runtime bundle id straight from pbxproj (the overlay step
# has already rewritten it to the elizaOS-specific id).
APP_ID=$(grep -m1 -E 'PRODUCT_BUNDLE_IDENTIFIER = [^.]+\.' ios/App/App.xcodeproj/project.pbxproj \
| grep -v WebsiteBlockerContentExtension \
| awk '{print $3}' | tr -d ';')
if [[ -z "$APP_ID" ]]; then
echo "::error::Could not resolve iOS bundle id from pbxproj"
exit 1
fi
APP_ID_EXTRA="${APP_ID}.WebsiteBlockerContentExtension"
echo "APP_IDENTIFIER=$APP_ID" >> "$GITHUB_ENV"
echo "APP_IDENTIFIER_EXTRA=$APP_ID_EXTRA" >> "$GITHUB_ENV"
echo "Resolved: APP_IDENTIFIER=$APP_ID, APP_IDENTIFIER_EXTRA=$APP_ID_EXTRA"
- name: Run iOS store preflight
working-directory: packages/app
run: bun run preflight:ios:store
- name: Build and upload
working-directory: packages/app/ios
env:
APP_IDENTIFIER: ${{ env.APP_IDENTIFIER }}
APP_IDENTIFIER_EXTRA: ${{ env.APP_IDENTIFIER_EXTRA }}
run: |
if [[ ! -f fastlane/Fastfile ]]; then
echo "::error::fastlane/Fastfile missing at packages/app/ios/. cap:sync:ios or run-mobile-build.mjs should have copied it."
ls -la fastlane/ || true
exit 1
fi
TRACK="${{ inputs.track || 'testflight' }}"
if [[ "$TRACK" == "app-store" ]]; then
bundle exec fastlane release
else
bundle exec fastlane beta
fi
- name: Upload IPA artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: ios-ipa
path: packages/app/ios/build/Eliza.ipa
retention-days: 90
if-no-files-found: error
# ── Mac App Store ────────────────────────────────────────────────────
build-macos:
name: Build & Submit macOS
needs: prepare
if: needs.prepare.outputs.build_macos == 'true'
runs-on: macos-15
timeout-minutes: 75
env:
APPLE_ID: ${{ secrets.APPLE_ID }}
APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
APPLE_APP_SPECIFIC_PASSWORD: ${{ secrets.APPLE_APP_SPECIFIC_PASSWORD }}
CSC_LINK: ${{ secrets.MAS_CSC_LINK }}
CSC_KEY_PASSWORD: ${{ secrets.MAS_CSC_KEY_PASSWORD }}
MAS_INSTALLER_CERT: ${{ secrets.MAS_INSTALLER_CERT }}
MAS_INSTALLER_KEY_PASSWORD: ${{ secrets.MAS_INSTALLER_KEY_PASSWORD }}
APP_STORE_API_KEY_ID: ${{ secrets.APP_STORE_API_KEY_ID }}
APP_STORE_API_ISSUER_ID: ${{ secrets.APP_STORE_API_ISSUER_ID }}
APP_STORE_API_KEY_P8: ${{ secrets.APP_STORE_API_KEY_P8 }}
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install dependencies
run: bun install --ignore-scripts || bun install --ignore-scripts
- name: Materialize bun npm package binary
run: if [ -f node_modules/bun/install.js ]; then (cd node_modules/bun && node install.js); fi
- name: Set version
env:
RELEASE_VERSION: ${{ needs.prepare.outputs.version }}
run: |
node -e "
const fs = require('fs');
const v = process.env.RELEASE_VERSION;
for (const f of ['package.json', 'packages/app/package.json', 'packages/app-core/platforms/electrobun/package.json']) {
try {
const pkg = JSON.parse(fs.readFileSync(f, 'utf8'));
pkg.version = v;
fs.writeFileSync(f, JSON.stringify(pkg, null, 2) + '\n');
} catch(e) {}
}
const cfgPath = 'packages/app-core/platforms/electrobun/electrobun.config.ts';
let cfg = fs.readFileSync(cfgPath, 'utf8');
cfg = cfg.replace(/version:\s*\"[^\"]+\"/, 'version: \"' + v + '\"');
fs.writeFileSync(cfgPath, cfg);
"
- name: Build core dist
run: |
bunx tsdown
echo '{"type":"module"}' > dist/package.json
node --import tsx packages/scripts/write-build-info.ts
- name: Build renderer
working-directory: packages/app
run: |
bun install --ignore-scripts
if [ -f node_modules/bun/install.js ]; then (cd node_modules/bun && node install.js); fi
bun run build:web
- name: Stage renderer
run: |
node packages/scripts/rm-path-recursive.mjs packages/app-core/platforms/electrobun/renderer
cp -r packages/app/dist packages/app-core/platforms/electrobun/renderer
- name: Build native macOS effects
working-directory: packages/app-core/platforms/electrobun
run: bun run build:native-effects
- name: Build webview preload
working-directory: packages/app-core/platforms/electrobun
run: bun run build:preload
- name: Validate signing secrets
run: |
missing=()
[[ -z "$CSC_LINK" ]] && missing+=("MAS_CSC_LINK")
[[ -z "$CSC_KEY_PASSWORD" ]] && missing+=("MAS_CSC_KEY_PASSWORD")
[[ -z "$MAS_INSTALLER_CERT" ]] && missing+=("MAS_INSTALLER_CERT")
[[ -z "$MAS_INSTALLER_KEY_PASSWORD" ]] && missing+=("MAS_INSTALLER_KEY_PASSWORD")
[[ -z "$APP_STORE_API_KEY_ID" ]] && missing+=("APP_STORE_API_KEY_ID")
[[ -z "$APP_STORE_API_ISSUER_ID" ]] && missing+=("APP_STORE_API_ISSUER_ID")
[[ -z "$APP_STORE_API_KEY_P8" ]] && missing+=("APP_STORE_API_KEY_P8")
if [[ ${#missing[@]} -gt 0 ]]; then
echo "::error::Missing required signing secrets: ${missing[*]}"
echo "Configure these secrets in the repo settings to enable Mac App Store builds."
exit 1
fi
- name: Import signing certificates
run: |
KEYCHAIN="build-mas.keychain"
KEYCHAIN_PASS="$(openssl rand -hex 16)"
security create-keychain -p "$KEYCHAIN_PASS" "$KEYCHAIN"
security default-keychain -s "$KEYCHAIN"
security unlock-keychain -p "$KEYCHAIN_PASS" "$KEYCHAIN"
security set-keychain-settings -t 3600 "$KEYCHAIN"
echo "$CSC_LINK" | base64 -d > /tmp/mas-app.p12
security import /tmp/mas-app.p12 -k "$KEYCHAIN" -P "$CSC_KEY_PASSWORD" \
-T /usr/bin/codesign -T /usr/bin/productbuild
rm -f /tmp/mas-app.p12
echo "$MAS_INSTALLER_CERT" | base64 -d > /tmp/mas-installer.p12
security import /tmp/mas-installer.p12 -k "$KEYCHAIN" -P "$MAS_INSTALLER_KEY_PASSWORD" \
-T /usr/bin/productbuild
rm -f /tmp/mas-installer.p12
security set-key-partition-list -S apple-tool:,apple:,codesign: -s -k "$KEYCHAIN_PASS" "$KEYCHAIN"
MAS_IDENTITY=$(security find-identity -v -p codesigning "$KEYCHAIN" | grep "Apple Distribution" | head -1 | awk -F'"' '{print $2}')
INSTALLER_IDENTITY=$(security find-identity -v "$KEYCHAIN" | grep "3rd Party Mac Developer Installer" | head -1 | awk -F'"' '{print $2}')
echo "MAS_IDENTITY=$MAS_IDENTITY" >> "$GITHUB_ENV"
echo "INSTALLER_IDENTITY=$INSTALLER_IDENTITY" >> "$GITHUB_ENV"
- name: Build Electrobun app
working-directory: packages/app-core/platforms/electrobun
run: ELECTROBUN_SKIP_CODESIGN=1 bun run build
env:
ELECTROBUN_BUILD_ENV: "stable"
- name: Re-sign for Mac App Store
working-directory: packages/app-core/platforms/electrobun
run: |
APP_PATH=$(find build -name "*.app" -maxdepth 2 | head -1)
ENTITLEMENTS="${GITHUB_WORKSPACE}/packages/app-core/platforms/electrobun/entitlements/mas.entitlements"
CHILD_ENTITLEMENTS="${GITHUB_WORKSPACE}/packages/app-core/platforms/electrobun/entitlements/mas-child.entitlements"
# Sign frameworks and dylibs with child entitlements
find "$APP_PATH" -name "*.dylib" -o -name "*.framework" | while read -r lib; do
codesign --force --options runtime --sign "$MAS_IDENTITY" \
--entitlements "$CHILD_ENTITLEMENTS" "$lib"
done
# Sign helper apps
find "$APP_PATH/Contents/Frameworks" -name "*.app" 2>/dev/null | while read -r helper; do
codesign --force --deep --options runtime --sign "$MAS_IDENTITY" \
--entitlements "$CHILD_ENTITLEMENTS" "$helper"
done
# Sign main app bundle
codesign --force --deep --options runtime --sign "$MAS_IDENTITY" \
--entitlements "$ENTITLEMENTS" "$APP_PATH"
codesign --verify --deep --strict "$APP_PATH"
- name: Create installer pkg
working-directory: packages/app-core/platforms/electrobun
run: |
APP_PATH=$(find build -name "*.app" -maxdepth 2 | head -1)
VERSION="${{ needs.prepare.outputs.version }}"
PKG_NAME="Eliza-${VERSION}-mas.pkg"
productbuild --sign "$INSTALLER_IDENTITY" \
--component "$APP_PATH" /Applications \
"build/$PKG_NAME"
echo "PKG_PATH=packages/app-core/platforms/electrobun/build/$PKG_NAME" >> "$GITHUB_ENV"
- name: Upload to App Store Connect
run: |
# xcrun altool requires the .p8 private key at the canonical path:
# ~/.appstoreconnect/private_keys/AuthKey_<KEY_ID>.p8
# The key content is provided via the APP_STORE_API_KEY_P8 secret.
KEY_DIR="$HOME/.appstoreconnect/private_keys"
mkdir -p "$KEY_DIR"
KEY_FILE="$KEY_DIR/AuthKey_${{ secrets.APP_STORE_API_KEY_ID }}.p8"
printf '%s' "$APP_STORE_API_KEY_P8" > "$KEY_FILE"
chmod 600 "$KEY_FILE"
xcrun altool --upload-app \
--file "$PKG_PATH" \
--type macos \
--apiKey "${{ secrets.APP_STORE_API_KEY_ID }}" \
--apiIssuer "${{ secrets.APP_STORE_API_ISSUER_ID }}" \
--show-progress
rm -f "$KEY_FILE"
- name: Upload pkg artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: macos-mas-pkg
path: ${{ env.PKG_PATH }}
retention-days: 90
if-no-files-found: error
# ── Summary ──────────────────────────────────────────────────────────
summary:
name: Release Summary
needs: [prepare, build-ios, build-macos]
if: always()
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- name: Summary
run: |
{
echo "## Apple Store Release Summary"
echo ""
echo "**Version:** ${{ needs.prepare.outputs.version }}"
echo "**Build:** ${{ needs.prepare.outputs.build_number }}"
echo ""
echo "| Platform | Status |"
echo "|----------|--------|"
echo "| iOS App Store | ${{ needs.build-ios.result || 'skipped' }} |"
echo "| Mac App Store | ${{ needs.build-macos.result || 'skipped' }} |"
} >> "$GITHUB_STEP_SUMMARY"
- name: Require enabled Apple releases succeeded
run: |
failed=0
if [[ "${{ needs.prepare.outputs.build_ios }}" == "true" && "${{ needs.build-ios.result }}" != "success" ]]; then
echo "::error::iOS release result was '${{ needs.build-ios.result }}', expected success"
failed=1
fi
if [[ "${{ needs.prepare.outputs.build_macos }}" == "true" && "${{ needs.build-macos.result }}" != "success" ]]; then
echo "::error::Mac App Store release result was '${{ needs.build-macos.result }}', expected success"
failed=1
fi
exit "$failed"
+89
View File
@@ -0,0 +1,89 @@
name: Apps Deploy E2E (Product 2)
# Hermetic, real-Docker end-to-end proof of the Apps (Product 2) deploy chain:
# build user image (real AppImageBuilder) -> provision a per-tenant Postgres DB
# -> run the image as an --internal isolated container with its per-tenant DSN
# -> it reaches ITS OWN DB and serves -> it is REJECTED from another tenant's DB
# (REVOKE CONNECT) -> it has NO internet egress.
#
# Self-contained: needs only docker + bun on the runner (no staging, no secrets,
# no VPS). Wires the previously-unwired verify scripts in
# packages/cloud/shared/scripts/ into CI so the deploy + per-tenant-DB + isolation
# mechanics are GATED, not manually run on a box — matching the "everything is
# CI/CD, no manual" model. Dispatchable on demand and gating on apps-deploy code.
on:
workflow_dispatch:
pull_request:
branches: [main]
paths:
- "packages/cloud/shared/src/lib/services/app-deploy-runner.ts"
- "packages/cloud/shared/src/lib/services/app-container-provider.ts"
- "packages/cloud/shared/src/lib/services/app-db-ambassador.ts"
- "packages/cloud/shared/src/lib/services/app-image-builder.ts"
- "packages/cloud/shared/src/lib/services/app-image-resolver.ts"
- "packages/cloud/shared/src/lib/services/container-executor-deps.ts"
- "packages/cloud/shared/src/lib/services/tenant-db/**"
- "packages/cloud/shared/scripts/verify-e2e-deploy.sh"
- "packages/cloud/shared/scripts/verify-e2e-container-db.sh"
- "packages/cloud/shared/scripts/verify-tenant-db-isolation.ts"
- "packages/cloud/shared/scripts/_e2e-build.ts"
- "packages/cloud/shared/scripts/e2e-sample-app/**"
- ".github/workflows/apps-deploy-e2e.yml"
permissions:
contents: read
concurrency:
group: apps-deploy-e2e-${{ github.ref }}
cancel-in-progress: true
jobs:
apps-deploy-e2e:
name: Apps deploy + per-tenant DB isolation (real docker)
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 25
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
- uses: ./.github/actions/cloud-setup-test-env
- name: Build cloud-shared (drizzle artifacts)
run: bun run --cwd packages/cloud/shared build || true
- name: E2E — build image → tenant DB → isolated container serves its own DB, blocked from others + egress
working-directory: packages/cloud/shared
run: bash scripts/verify-e2e-deploy.sh
- name: E2E — real-docker tenant-DB isolation (REVOKE CONNECT + --internal no-egress)
working-directory: packages/cloud/shared
run: bash scripts/verify-e2e-container-db.sh
tenant-db-isolation:
name: Tenant-DB isolation — cross-tenant REVOKE CONNECT (throwaway Postgres) (#9853)
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 15
# Lightweight pg-only proof (no docker): provision two tenants through the
# real SqlTenantDbProvisioner against a throwaway Postgres and assert tenant
# A cannot connect to tenant B's database. Wires the previously-orphaned
# verify-tenant-db-isolation.ts into CI.
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_PASSWORD: adminpw
ports:
- 55432:5432
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 5s
--health-timeout 5s
--health-retries 20
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
- uses: ./.github/actions/cloud-setup-test-env
- name: Verify cross-tenant REVOKE CONNECT on per-tenant databases
working-directory: packages/cloud/shared
run: bun run verify:tenant-isolation
+218
View File
@@ -0,0 +1,218 @@
name: Arm Apps Daemon (Product 2)
# Arm the apps (Product 2) deploy backend on a provisioning-worker control plane
# WITHOUT hand-editing the box — the IaC counterpart to hand-SSH'ing in.
#
# The daemon already calls configureAppsDeployBackend() on boot, but it only
# provisions app containers once the apps env is present in
# /opt/eliza/cloud/.env.local. This workflow UPSERTS exactly that block (each
# key delete-then-append, every other line untouched — never clobbers the
# agent/coding fleet config) and restarts the daemon, over the SAME
# ELIZA_PROVISIONING_SSH_KEY/HOST secrets the deploy workflow already uses. So
# no operator SSH key needs to be added anywhere; dispatch it and it converges.
#
# Mirrors packages/scripts/cloud/admin/arm-apps-daemon.mjs (same upsert logic),
# so a local run and a CI run produce the same box state.
#
# Inputs:
# environment staging | production (selects the env's PROVISIONING secrets)
# app_node CONTAINERS_DOCKER_NODES value, e.g. apps-node-1:167.233.112.155:20
# tenant_admin_dsn (optional) env-sourced tenant DB admin DSN. Leave empty to
# let the daemon use the SEEDED tenant_db_clusters row
# (encrypted path). Set it only if the encrypted path isn't
# wired on this box.
# egress_proxy (optional) CONTAINERS_EGRESS_PROXY_URL
#
# Non-secret base domain comes from the env var APPS_BASE_DOMAIN (same var the
# terraform workflow reads); the Caddy admin URL is derived from the app node IP.
on:
workflow_dispatch:
inputs:
environment:
description: Environment to target
type: choice
default: staging
options: [staging, production]
target:
description: "Which daemon to arm: the shared control-plane provisioning-worker (cp-daemon) or the dedicated apps-control worker (apps-worker, Product-2 isolated path)"
type: choice
default: cp-daemon
options: [cp-daemon, apps-worker]
app_node:
description: "CONTAINERS_DOCKER_NODES (id:ip:capacity), e.g. apps-node-1:167.233.112.155:20"
type: string
required: true
tenant_admin_dsn:
description: "Optional env-sourced tenant DB admin DSN (leave empty to use the seeded encrypted cluster OR fetch_dsn_from_shared_state)"
type: string
required: false
default: ""
fetch_dsn_from_shared_state:
description: "Read APPS_TENANT_ADMIN_DSN from the apps-shared terraform output instead of pasting it (keeps the password out of inputs/logs)"
type: boolean
required: false
default: false
egress_proxy:
description: "Optional CONTAINERS_EGRESS_PROXY_URL"
type: string
required: false
default: ""
permissions:
contents: read
concurrency:
# Key on target too: the cp-daemon and apps-worker arms touch different hosts,
# so serializing them against each other (env-only key) needlessly blocks
# arming the two independent daemons concurrently.
group: arm-apps-daemon-${{ github.event.inputs.environment }}-${{ github.event.inputs.target }}
cancel-in-progress: false
jobs:
arm:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
environment: ${{ github.event.inputs.environment }}
timeout-minutes: 10
env:
# Target the dedicated apps-control host (apps-worker) or the shared
# control-plane host (cp-daemon). The apps-worker path arms the isolated
# Product-2 daemon on a host that sits on the apps private net (so it can
# reach the tenant DB) and runs no agent jobs — never touching the agent
# control plane.
#
# NO cross-target fallback. The naive `apps-worker && APPS || PROVISIONING`
# form silently resolves to the CP host when ELIZA_APPS_WORKER_HOST is
# unset — which would SSH the apps env onto the agent control-plane host
# (Preflight can't catch it because DEPLOY_HOST is then non-empty). The
# nested form below only ever yields the SAME-target secret or '' (the
# trailing `|| ''` forces empty, not the boolean `false`), so a missing
# same-target secret leaves DEPLOY_HOST empty and the Preflight
# `[ -n "$DEPLOY_HOST" ]` check reports the right missing-secret error.
DEPLOY_HOST: ${{ (github.event.inputs.target == 'apps-worker' && secrets.ELIZA_APPS_WORKER_HOST) || (github.event.inputs.target == 'cp-daemon' && secrets.ELIZA_PROVISIONING_HOST) || '' }}
DEPLOY_SSH_KEY: ${{ (github.event.inputs.target == 'apps-worker' && secrets.ELIZA_APPS_WORKER_SSH_KEY) || (github.event.inputs.target == 'cp-daemon' && secrets.ELIZA_PROVISIONING_SSH_KEY) || '' }}
CONTAINERS_SSH_KEY_INPUT: ${{ secrets.CONTAINERS_SSH_KEY || (github.event.inputs.target == 'apps-worker' && secrets.ELIZA_APPS_WORKER_SSH_KEY) || (github.event.inputs.target == 'cp-daemon' && secrets.ELIZA_PROVISIONING_SSH_KEY) || '' }}
SYSTEMD_UNIT: ${{ github.event.inputs.target == 'apps-worker' && 'eliza-apps-worker.service' || 'eliza-provisioning-worker.service' }}
BASE_DOMAIN: ${{ vars.APPS_BASE_DOMAIN }}
APP_NODE: ${{ github.event.inputs.app_node }}
TENANT_ADMIN_DSN: ${{ github.event.inputs.tenant_admin_dsn }}
EGRESS_PROXY: ${{ github.event.inputs.egress_proxy }}
steps:
- name: Preflight
run: |
fail=0
HOST_SECRET="${{ github.event.inputs.target == 'apps-worker' && 'ELIZA_APPS_WORKER_HOST' || 'ELIZA_PROVISIONING_HOST' }}"
KEY_SECRET="${{ github.event.inputs.target == 'apps-worker' && 'ELIZA_APPS_WORKER_SSH_KEY' || 'ELIZA_PROVISIONING_SSH_KEY' }}"
[ -n "$DEPLOY_HOST" ] || { echo "::error::missing secret $HOST_SECRET on the '${{ github.event.inputs.environment }}' environment (target=${{ github.event.inputs.target }})"; fail=1; }
[ -n "$DEPLOY_SSH_KEY" ] || { echo "::error::missing secret $KEY_SECRET"; fail=1; }
[ -n "$CONTAINERS_SSH_KEY_INPUT" ] || { echo "::error::missing container SSH key material: set secret CONTAINERS_SSH_KEY or $KEY_SECRET"; fail=1; }
[ -n "$BASE_DOMAIN" ] || { echo "::error::set vars.APPS_BASE_DOMAIN on the '${{ github.event.inputs.environment }}' environment"; fail=1; }
[ -n "$APP_NODE" ] || { echo "::error::app_node input is required (id:ip:capacity)"; fail=1; }
# Derive the Caddy admin URL from the app node IP (2nd colon field).
NODE_IP="$(echo "$APP_NODE" | cut -d: -f2)"
echo "$NODE_IP" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$' || { echo "::error::app_node must be id:ip:capacity with a valid IPv4"; fail=1; }
echo "CADDY_ADMIN=http://$NODE_IP:2019" >> "$GITHUB_ENV"
[ "$fail" = 0 ] || exit 1
- name: Prepare container SSH key
run: |
set -euo pipefail
tmp="$(mktemp)"
cleanup() { rm -f "$tmp"; }
trap cleanup EXIT
# docker-ssh.ts consumes CONTAINERS_SSH_KEY as base64. Accept either
# a dedicated already-base64 CONTAINERS_SSH_KEY secret or the raw
# target deploy key used by appleboy/ssh-action, then forward one
# single-line base64 value to the remote upsert script.
if printf '%s' "$CONTAINERS_SSH_KEY_INPUT" | base64 -d >"$tmp" 2>/dev/null && grep -q 'BEGIN .*PRIVATE KEY' "$tmp"; then
containers_ssh_key_b64="$(printf '%s' "$CONTAINERS_SSH_KEY_INPUT" | tr -d '\r\n')"
else
containers_ssh_key_b64="$(printf '%s' "$CONTAINERS_SSH_KEY_INPUT" | base64 -w0)"
fi
[ -n "$containers_ssh_key_b64" ] || { echo "::error::container SSH key resolved to an empty value"; exit 1; }
echo "::add-mask::$containers_ssh_key_b64"
echo "CONTAINERS_SSH_KEY_B64=$containers_ssh_key_b64" >> "$GITHUB_ENV"
# OPTIONAL: pull the tenant admin DSN straight from the apps-shared
# terraform output so the password never appears in a workflow input or in
# logs. `terraform output -raw` of a SENSITIVE value is auto-masked by the
# runner; we also ::add-mask:: it before writing to $GITHUB_ENV.
- name: Checkout (for terraform fetch)
if: ${{ github.event.inputs.fetch_dsn_from_shared_state == 'true' }}
uses: actions/checkout@v4
- name: Setup Terraform (for DSN fetch)
if: ${{ github.event.inputs.fetch_dsn_from_shared_state == 'true' }}
uses: hashicorp/setup-terraform@v4
with:
terraform_version: "1.10.5"
- name: Read DSN from apps-shared output
if: ${{ github.event.inputs.fetch_dsn_from_shared_state == 'true' }}
working-directory: packages/cloud/infra/cloud/terraform/hetzner/apps-shared
env:
HCLOUD_TOKEN: ${{ secrets.HCLOUD_APPS_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_STATE_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_STATE_SECRET_ACCESS_KEY }}
run: |
terraform init -input=false -backend-config=backend.hcl >/dev/null
DSN="$(terraform output -raw tenant_db_admin_dsn)"
[ -n "$DSN" ] || { echo "::error::apps-shared has no tenant_db_admin_dsn output (is it applied?)"; exit 1; }
echo "::add-mask::$DSN"
echo "TENANT_ADMIN_DSN=$DSN" >> "$GITHUB_ENV"
echo "fetched tenant admin DSN from apps-shared output (masked)"
- name: Upsert apps env + restart daemon
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2
with:
host: ${{ env.DEPLOY_HOST }}
username: deploy
key: ${{ env.DEPLOY_SSH_KEY }}
command_timeout: 5m
# Only the values; the upsert logic is fixed here (mirrors arm-apps-daemon.mjs).
envs: APP_NODE,BASE_DOMAIN,CADDY_ADMIN,TENANT_ADMIN_DSN,EGRESS_PROXY,SYSTEMD_UNIT,CONTAINERS_SSH_KEY_B64
script: |
set -euo pipefail
F=/opt/eliza/cloud/.env.local
# /opt/eliza/cloud is root-owned; the deploy user has passwordless
# sudo (same as the worker-deploy workflow), so all writes go via
# sudo — `sed -i` needs to create a temp file IN the dir, not just
# write the file, which a non-sudo deploy user can't do there.
sudo test -f "$F" || { echo "env file $F not found on host"; exit 1; }
sudo cp -n "$F" "$F.bak.arm-apps" 2>/dev/null || true
upsert() {
# upsert KEY VALUE — delete any existing def, append the fresh one.
local k="$1"; local v="$2"
[ -n "$v" ] || return 0
sudo sed -i "/^$k=/d" "$F"
printf '%s\n' "$k=\"$v\"" | sudo tee -a "$F" >/dev/null
}
# APPS_DEPLOY_ENABLED arms the APP_DEPLOY runner; without it the daemon's
# armAppsDeployBackendIfEnabled() early-returns (provisioning-worker.ts:652)
# and APP_DEPLOY jobs never get claimed (apps stuck "building").
upsert APPS_DEPLOY_ENABLED 1
upsert APPS_CONTAINERS_ENABLED 1
upsert CONTAINERS_DOCKER_NODES "$APP_NODE"
upsert CONTAINERS_SSH_USER deploy
upsert CONTAINERS_SSH_KEY "$CONTAINERS_SSH_KEY_B64"
upsert APPS_CADDY_ADMIN_URL "$CADDY_ADMIN"
upsert CONTAINERS_PUBLIC_BASE_DOMAIN "$BASE_DOMAIN"
upsert APPS_TENANT_ADMIN_DSN "$TENANT_ADMIN_DSN"
upsert CONTAINERS_EGRESS_PROXY_URL "$EGRESS_PROXY"
echo "--- apps env now on the box (secrets redacted) ---"
sudo grep -E '^(APPS_|CONTAINERS_(DOCKER_NODES|SSH_USER|SSH_KEY|PUBLIC_BASE_DOMAIN|EGRESS_PROXY_URL))' "$F" \
| sed -E 's/(DSN|KEY)=.*/\1=<redacted>/'
# Restart whichever daemon we armed: the dedicated apps-worker on the
# apps-control host, or the shared control-plane provisioning-worker.
UNIT="${SYSTEMD_UNIT:-eliza-provisioning-worker.service}"
sudo systemctl restart "$UNIT"
sleep 2
echo "--- daemon status ($UNIT) ---"
systemctl is-active "$UNIT"
journalctl -u "$UNIT" -n 10 --no-pager | tail -10 || true
@@ -0,0 +1,112 @@
name: Arm Headscale Control Plane
# Converges Headscale on the Hetzner control-plane host. This is the
# repeatable version of the launch runbook: configure server_url/listen_addr,
# install the committed ACL, ensure users exist, upsert daemon env, restart
# headscale + provisioning-worker, and fail if local /health is not green.
#
# WHY: agent provisioning must not be allowed to "sort of" work. If the daemon
# has a Headscale key, docker-sandbox-provider requires a real headscale_ip
# before marking a sandbox running. This workflow makes that dependency explicit
# and keeps prod/staging wiring in GitHub Environments instead of hand-edited
# shell history on the host.
on:
workflow_dispatch:
inputs:
environment:
description: "Target control-plane environment"
type: choice
default: staging
options: [staging, production]
headscale_public_url:
description: "Public Headscale URL. Empty = vars.HEADSCALE_PUBLIC_URL"
type: string
required: false
default: ""
headscale_api_url:
description: "Daemon-local Headscale API URL"
type: string
required: false
default: "http://127.0.0.1:8081"
listen_addr:
description: "Headscale listen_addr on the CP"
type: string
required: false
default: "127.0.0.1:8081"
permissions:
contents: read
concurrency:
group: arm-headscale-control-plane-${{ github.event.inputs.environment }}
cancel-in-progress: false
jobs:
arm:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
environment: ${{ github.event.inputs.environment }}
timeout-minutes: 10
env:
DEPLOY_HOST: ${{ secrets.ELIZA_PROVISIONING_HOST }}
DEPLOY_SSH_KEY: ${{ secrets.ELIZA_PROVISIONING_SSH_KEY }}
HEADSCALE_API_KEY: ${{ secrets.HEADSCALE_API_KEY }}
AGENT_TOKEN_PRIVATE_KEY_PEM: ${{ secrets.AGENT_TOKEN_PRIVATE_KEY_PEM }}
ELIZA_LOCAL_ROOT_KEY: ${{ secrets.ELIZA_LOCAL_ROOT_KEY }}
HEADSCALE_PUBLIC_URL_INPUT: ${{ github.event.inputs.headscale_public_url }}
HEADSCALE_PUBLIC_URL_VAR: ${{ vars.HEADSCALE_PUBLIC_URL }}
HEADSCALE_API_URL: ${{ github.event.inputs.headscale_api_url }}
HEADSCALE_LISTEN_ADDR: ${{ github.event.inputs.listen_addr }}
# Let's Encrypt account / expiry-notice email for the headscale vhost cert
# the arm script provisions. Optional GitHub Environment var; the script
# falls back to ops@elizalabs.ai when unset (so existing envs need no new
# var to keep working).
CERTBOT_EMAIL: ${{ vars.CERTBOT_EMAIL }}
steps:
- uses: actions/checkout@v4
- name: Preflight
run: |
fail=0
[ -n "$DEPLOY_HOST" ] || { echo "::error::missing secret ELIZA_PROVISIONING_HOST"; fail=1; }
[ -n "$DEPLOY_SSH_KEY" ] || { echo "::error::missing secret ELIZA_PROVISIONING_SSH_KEY"; fail=1; }
[ -n "$HEADSCALE_API_KEY" ] || { echo "::error::missing secret HEADSCALE_API_KEY"; fail=1; }
if [ -n "$HEADSCALE_PUBLIC_URL_INPUT" ]; then
echo "HEADSCALE_PUBLIC_URL=$HEADSCALE_PUBLIC_URL_INPUT" >> "$GITHUB_ENV"
elif [ -n "$HEADSCALE_PUBLIC_URL_VAR" ]; then
echo "HEADSCALE_PUBLIC_URL=$HEADSCALE_PUBLIC_URL_VAR" >> "$GITHUB_ENV"
else
echo "::error::set vars.HEADSCALE_PUBLIC_URL or provide headscale_public_url"
fail=1
fi
[ "$fail" = 0 ] || exit 1
- name: Write SSH key
run: |
install -d -m 700 "$RUNNER_TEMP/headscale"
printf '%s\n' "$DEPLOY_SSH_KEY" > "$RUNNER_TEMP/headscale/deploy_ed25519"
chmod 600 "$RUNNER_TEMP/headscale/deploy_ed25519"
echo "DEPLOY_SSH_KEY_PATH=$RUNNER_TEMP/headscale/deploy_ed25519" >> "$GITHUB_ENV"
- name: Arm Headscale and daemon env
run: |
node packages/scripts/cloud/admin/arm-headscale-control-plane.mjs \
--host "$DEPLOY_HOST" \
--ssh-key "$DEPLOY_SSH_KEY_PATH" \
--headscale-public-url "$HEADSCALE_PUBLIC_URL" \
--headscale-api-url "$HEADSCALE_API_URL" \
--listen-addr "$HEADSCALE_LISTEN_ADDR"
- name: Summary
run: |
{
echo "### Headscale control plane armed"
echo ""
echo "- Environment: \`${{ github.event.inputs.environment }}\`"
echo "- Public URL: \`$HEADSCALE_PUBLIC_URL\`"
echo "- Daemon API URL: \`$HEADSCALE_API_URL\`"
echo ""
echo "Also converged on the CP (idempotent): nginx vhost + Let's Encrypt cert for the public headscale URL, and the cp-<env>-router tailscale self-enrollment (tag:eliza-proxy)."
echo ""
echo "Next: set matching Cloudflare Worker secrets, then run the provision E2E."
} >> "$GITHUB_STEP_SUMMARY"
@@ -0,0 +1,60 @@
name: Attachment journey videos
# Manual-only (workflow_dispatch): records full video runthroughs of the
# attachment user journeys — the Files view (load + facets + CRUD affordances)
# and the chat attachment upload flow — and uploads the .webm recordings +
# traces + screenshots as artifacts (E2E_RECORD=1 → e2e-recordings/). Kept off
# PR/develop CI to keep those runs fast; trigger from the Actions tab when a
# journey video is wanted. Runs on top of the keyless ui-smoke stub stack
# (no secrets), and the renderer build picks up @elizaos/core via the
# core-build pre-step in run-ui-playwright.mjs.
on:
workflow_dispatch:
env:
BUN_VERSION: "canary"
NODE_VERSION: "24.15.0"
jobs:
attachment-journey-videos:
name: Attachment journey videos
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 40
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install --ignore-scripts --no-frozen-lockfile
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Ensure generated shared i18n data
run: node packages/app-core/scripts/ensure-shared-i18n-data.mjs
- name: Install Playwright browsers
run: bunx playwright install --with-deps chromium
- name: Record attachment journeys (Files view + chat attachment)
run: bun run --cwd packages/app test:e2e:record test/ui-smoke/files-view.spec.ts test/ui-smoke/chat-attachment.spec.ts --project=chromium
- name: Upload journey videos + traces
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: attachment-journey-videos
path: |
e2e-recordings/
packages/app/playwright-report/
if-no-files-found: warn
retention-days: 14
+70
View File
@@ -0,0 +1,70 @@
name: Auto Label
on:
pull_request_target:
types: [opened, synchronize]
issues:
types: [opened, edited]
# #14051 Tier B: label runs are pure metadata — a superseded label pass on the
# same PR/issue is worthless. Cancel the in-flight run when a new event lands so
# rapid pushes don't pile up 70+ queued Auto Label runs on the fleet. Group by
# PR number (or issue number) so PR and issue lanes never collide.
concurrency:
group: auto-label-${{ github.event.pull_request.number || github.event.issue.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
pull-requests: write
issues: write
jobs:
label-pr:
if: github.event_name == 'pull_request_target'
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- uses: actions/labeler@f27b608878404679385c85cfa523b85ccb86e213
with:
repo-token: "${{ secrets.GITHUB_TOKEN }}"
pr-number: "${{ github.event.pull_request.number }}"
label-issue:
if: github.event_name == 'issues'
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3
with:
script: |
const title = context.payload.issue.title.toLowerCase();
const body = (context.payload.issue.body || '').toLowerCase();
const text = title + ' ' + body;
const labels = [];
if (text.includes('crash') || text.includes('error') || text.includes('bug') || text.includes('broken') || text.includes('regression')) {
labels.push('bug');
}
if (text.includes('security') || text.includes('vulnerability') || text.includes('cve') || text.includes('injection')) {
labels.push('security');
}
if (text.includes('performance') || text.includes('slow') || text.includes('memory leak') || text.includes('latency')) {
labels.push('performance');
}
if (text.includes('telegram') || text.includes('discord') || text.includes('connector')) {
labels.push('connector');
}
if (text.includes('memory') || text.includes('context') || text.includes('rag')) {
labels.push('memory');
}
if (text.includes('plugin')) {
labels.push('plugin');
}
if (labels.length > 0) {
await github.rest.issues.addLabels({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
labels: labels
});
}
@@ -0,0 +1,154 @@
# Benchmark orchestrator — scheduled real-model lane
#
# Runs a core subset of registry benchmarks through the Python orchestrator on a
# REAL model provider (Cerebras gemma-4-31b), so the suite is actually exercised
# end-to-end on a schedule rather than only vitest-testing the bridge code
# (#9475). Before this lane, `python -m benchmarks.orchestrator run` was invoked
# by exactly one workflow (hyperliquid-bench-live.yml), leaving 42/43 registered
# benchmarks with zero scheduled real-model runs.
#
# OPERATOR PREREQUISITE: add a `CEREBRAS_API_KEY` repo secret. With no secret
# configured this workflow SKIPS GRACEFULLY — it never fails CI when the key is
# absent (mirrors hyperliquid-bench-live.yml's gate).
#
# The benchmark subset (bfcl, action-calling, agentbench, tau_bench, mint,
# context_bench) is the confirmed-real, bridge-wired core that routes through a
# real AgentRuntime + LLM plugins via the eliza-adapter.
#
# Required repo secret (self-documented):
# CEREBRAS_API_KEY — Cerebras API key for the real-model orchestrator run.
name: Benchmark orchestrator (scheduled)
on:
schedule:
# Mondays 06:00 UTC — a weekly real-model smoke of the core registry subset.
- cron: "0 6 * * 1"
workflow_dispatch:
inputs:
benchmarks:
description: "Comma-separated registry benchmark ids"
required: false
type: string
default: "bfcl,action-calling,agentbench,tau_bench,mint,context_bench"
provider:
description: "Model provider"
required: false
type: string
default: "cerebras"
model:
description: "Model name"
required: false
type: string
default: "gemma-4-31b"
concurrency:
group: benchmark-orchestrator-scheduled-${{ github.ref }}
cancel-in-progress: false
# Default to least privilege.
permissions:
contents: read
env:
PYTHON_VERSION: "3.12"
BUN_VERSION: "canary"
NODE_VERSION: "24"
jobs:
orchestrator-run:
name: orchestrator-run
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 120
env:
CEREBRAS_API_KEY: ${{ secrets.CEREBRAS_API_KEY }}
steps:
# ------------------------------------------------------------------
# Gate: skip everything (exit 0) when CEREBRAS_API_KEY is unset, so this
# workflow never fails CI when the secret is not configured. Mirrors the
# gate in hyperliquid-bench-live.yml.
# ------------------------------------------------------------------
- name: Check CEREBRAS_API_KEY secret
id: gate
run: |
set -euo pipefail
if [ -z "${CEREBRAS_API_KEY:-}" ]; then
echo "::notice::Benchmark orchestrator scheduled run skipped — CEREBRAS_API_KEY is not configured. Add it as a repo secret to enable real-model orchestrator runs."
echo "skip=true" >> "$GITHUB_OUTPUT"
else
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
- name: Checkout
if: steps.gate.outputs.skip != 'true'
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: recursive
fetch-depth: 1
- name: Setup Python
if: steps.gate.outputs.skip != 'true'
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Setup Node.js
if: steps.gate.outputs.skip != 'true'
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
if: steps.gate.outputs.skip != 'true'
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install --ignore-scripts --no-frozen-lockfile
# ------------------------------------------------------------------
# The orchestrator package is `benchmarks` under `packages/`, so the
# module is invoked as `python -m benchmarks.orchestrator` with
# `PYTHONPATH=packages` from the repo root.
#
# NOTE: `validate-runtime-gates` is intentionally NOT run here — it checks
# ALL live gates globally (Hyperliquid key, Docker backends, …) and exits
# non-zero when any is unavailable, which is unrelated to this LLM-only core
# subset. The subset-scoped publishability gate below is the relevant check.
# ------------------------------------------------------------------
- name: Run core registry subset on a real model
if: steps.gate.outputs.skip != 'true'
id: run
env:
BENCHMARKS: ${{ inputs.benchmarks || 'bfcl,action-calling,agentbench,tau_bench,mint,context_bench' }}
PROVIDER: ${{ inputs.provider || 'cerebras' }}
MODEL: ${{ inputs.model || 'gemma-4-31b' }}
PYTHONUNBUFFERED: "1"
run: |
set -e
PYTHONPATH=packages python -m benchmarks.orchestrator run \
--benchmarks "$BENCHMARKS" \
--provider "$PROVIDER" \
--model "$MODEL" \
--force \
--show-incompatible \
2>&1 | tee orchestrator-scheduled.log
- name: Validate latest publishability (rejects mock/sample/demo rows)
if: steps.gate.outputs.skip != 'true'
env:
PYTHONUNBUFFERED: "1"
run: |
set -e
PYTHONPATH=packages python -m benchmarks.orchestrator validate-latest-publishability \
--include-benchmarks "${{ inputs.benchmarks || 'bfcl,action-calling,agentbench,tau_bench,mint,context_bench' }}"
- name: Upload orchestrator artifacts
if: always() && steps.gate.outputs.skip != 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: benchmark-orchestrator-scheduled-${{ github.run_id }}
path: |
packages/benchmarks/benchmark_results/**
orchestrator-scheduled.log
if-no-files-found: warn
retention-days: 90
+70
View File
@@ -0,0 +1,70 @@
name: Benchmark Bridge Tests
# CI fan-out fix: the PR-scoped concurrency from #14069 fell back to
# github.run_id on push, so every develop push got a unique group and
# nothing ever superseded. A 37-merge wave queued thousands of push runs
# and starved the self-hosted fleet. Fall back to github.ref (all develop
# pushes share a group) and cancel superseded runs on push too. PR behavior
# is unchanged (pull_request.number still wins the key); merge_group and
# schedule events are not in the cancel set, so the merge queue and nightly
# runs always complete.
concurrency:
group: benchmark-tests-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }}
on:
push:
branches: [main, develop]
paths:
- "packages/lifeops-bench/**"
- ".github/workflows/benchmark-tests.yml"
pull_request:
branches: [main]
paths:
- "packages/lifeops-bench/**"
- ".github/workflows/benchmark-tests.yml"
workflow_dispatch:
env:
NODE_NO_WARNINGS: "1"
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
benchmark:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
include:
- lane: benchmark-tests
command: bunx vitest run --config packages/lifeops-bench/vitest.config.ts --root packages/lifeops-bench --passWithNoTests
- lane: benchmark-lint
command: bunx @biomejs/biome check packages/lifeops-bench/src
steps:
- name: Checkout repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24"
check-latest: false
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: "1.3.14" # pinned: floating canary writes lockfileVersion 2 and breaks --frozen-lockfile (#11184/#9454); packageManager bun@1.4.0-canary.1 is unresolvable (GH+npm 404)
install-command: bun install --ignore-scripts --no-frozen-lockfile
install-native-deps: "false"
- name: Run ${{ matrix.lane }} lane
env:
CI: "true"
run: ${{ matrix.command }}
+221
View File
@@ -0,0 +1,221 @@
# Weekly EA + Connector Benchmark
#
# Runs the full executive-assistant and connector certification scenario
# catalogs through the live scenario runner and uploads a markdown + JSON
# benchmark report as an artifact. Scheduled/default dispatch runs are
# report-only so the weekly trend capture stays green while the live catalog
# is still being hardened; manual dispatch can opt into enforcement.
#
# Distinct from live-scenarios.yml: weekly cadence, emits a human-readable
# markdown report suitable for tracking trend lines over time.
#
# Required repo secrets (self-documented): identical to live-scenarios.yml.
# LLM providers: OPENAI_API_KEY, OPENROUTER_API_KEY, ANTHROPIC_API_KEY,
# GOOGLE_GENERATIVE_AI_API_KEY, GOOGLE_API_KEY, GROQ_API_KEY
# Google: GOOGLE_OAUTH_CLIENT_ID, GOOGLE_OAUTH_CLIENT_SECRET,
# GOOGLE_OAUTH_REFRESH_TOKEN, GMAIL_TEST_ACCOUNT_EMAIL,
# GMAIL_TEST_ACCOUNT_REFRESH_TOKEN
# Calendly: CALENDLY_API_TOKEN
# Telegram: TELEGRAM_BOT_TOKEN, TELEGRAM_TEST_CHAT_ID, TELEGRAM_API_ID,
# TELEGRAM_API_HASH
# Discord: DISCORD_BOT_TOKEN, DISCORD_TEST_GUILD_ID,
# DISCORD_TEST_CHANNEL_ID
# Signal: SIGNAL_CLI_URL, SIGNAL_TEST_NUMBER
# iMessage: IMESSAGE_BRIDGE_URL, IMESSAGE_TEST_HANDLE
# WhatsApp: WHATSAPP_TOKEN, WHATSAPP_PHONE_ID, WHATSAPP_TEST_CONTACT
# Twilio: TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN, TWILIO_FROM_NUMBER,
# TWILIO_TEST_TO_NUMBER
# X: X_API_KEY, X_API_SECRET, X_ACCESS_TOKEN, X_ACCESS_SECRET,
# X_TEST_DM_HANDLE
# Push: NOTIFICATION_RELAY_URL, NOTIFICATION_RELAY_TOKEN
# Travel: TRAVEL_BOOKING_API_KEY
# Optional:
# LIFEOPS_JUDGE_THRESHOLD (workflow input, default 0.8)
# SCENARIO_FILTER (workflow input, empty = all cataloged benchmark scenarios)
# BENCHMARK_ENFORCE_GATE (workflow input, default false)
name: Benchmark (weekly)
on:
schedule:
# Mondays 10:00 UTC — after weekend regressions settle.
- cron: "0 10 * * 1"
workflow_dispatch:
inputs:
scenario_filter:
description: "Comma-separated scenario ids (empty = full benchmark catalog)"
required: false
type: string
default: ""
judge_threshold:
description: "LLM-judge minimum pass score (0.0 - 1.0)"
required: false
type: string
default: "0.8"
enforce_gate:
description: "Fail the workflow when benchmark scenarios fail"
required: false
type: boolean
default: false
concurrency:
group: benchmark-weekly-${{ github.ref }}
cancel-in-progress: false
env:
BUN_VERSION: "canary"
NODE_VERSION: "24.15.0"
ELIZA_LIVE_TEST: "1"
permissions:
contents: read
actions: read
jobs:
benchmark:
name: EA + connector benchmark
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 240
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 0
filter: blob:none
submodules: recursive
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install --ignore-scripts --no-frozen-lockfile
- name: Verify scenario-runner CLI present
run: |
test -f packages/scenario-runner/src/cli.ts || \
{ echo "scenario-runner CLI missing"; exit 1; }
- name: Build benchmark runtime packages
# The benchmark executes TypeScript sources directly, but several
# workspace packages intentionally export dist/* entry points. Because
# dependency installation ignores postinstall scripts, build only the
# packages that the live scenario runtime imports through those exports.
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_GENERATIVE_AI_API_KEY }}
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
run: |
echo "::group::Build packages/core"
bun run --cwd packages/core build
echo "::endgroup::"
echo "::group::Build packages/shared"
# plugin-agent-skills (and others) consume @elizaos/shared via dist
# exports (RouteRequestContext, route schemas). Must build before
# any plugin that imports from it.
bun run --cwd packages/shared build
echo "::endgroup::"
echo "::group::Build packages/skills"
# plugin-agent-skills imports @elizaos/skills via its dist exports.
bun run --cwd packages/skills build
echo "::endgroup::"
provider_package=""
if [ -n "${GROQ_API_KEY:-}" ]; then
provider_package="plugins/plugin-groq"
elif [ -n "${OPENAI_API_KEY:-}" ]; then
provider_package="plugins/plugin-openai"
elif [ -n "${ANTHROPIC_API_KEY:-}" ]; then
provider_package="plugins/plugin-anthropic"
elif [ -n "${GOOGLE_GENERATIVE_AI_API_KEY:-}" ] || [ -n "${GOOGLE_API_KEY:-}" ]; then
provider_package="plugins/plugin-google-genai"
elif [ -n "${OPENROUTER_API_KEY:-}" ]; then
provider_package="plugins/plugin-openrouter"
fi
package_dirs=(
plugins/plugin-sql
plugins/plugin-agent-skills
plugins/plugin-pdf
plugins/plugin-telegram
plugins/plugin-whatsapp
plugins/plugin-signal
plugins/plugin-imessage
)
if [ -n "$provider_package" ]; then
package_dirs+=("$provider_package")
fi
for package_dir in "${package_dirs[@]}"; do
echo "::group::Build ${package_dir}"
bun run --cwd "$package_dir" build
echo "::endgroup::"
done
- name: Run benchmark harness
id: run
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_GENERATIVE_AI_API_KEY }}
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
GOOGLE_OAUTH_CLIENT_ID: ${{ secrets.GOOGLE_OAUTH_CLIENT_ID }}
GOOGLE_OAUTH_CLIENT_SECRET: ${{ secrets.GOOGLE_OAUTH_CLIENT_SECRET }}
GOOGLE_OAUTH_REFRESH_TOKEN: ${{ secrets.GOOGLE_OAUTH_REFRESH_TOKEN }}
GMAIL_TEST_ACCOUNT_EMAIL: ${{ secrets.GMAIL_TEST_ACCOUNT_EMAIL }}
GMAIL_TEST_ACCOUNT_REFRESH_TOKEN: ${{ secrets.GMAIL_TEST_ACCOUNT_REFRESH_TOKEN }}
CALENDLY_API_TOKEN: ${{ secrets.CALENDLY_API_TOKEN }}
TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }}
TELEGRAM_TEST_CHAT_ID: ${{ secrets.TELEGRAM_TEST_CHAT_ID }}
TELEGRAM_API_ID: ${{ secrets.TELEGRAM_API_ID }}
TELEGRAM_API_HASH: ${{ secrets.TELEGRAM_API_HASH }}
DISCORD_BOT_TOKEN: ${{ secrets.DISCORD_BOT_TOKEN }}
DISCORD_TEST_GUILD_ID: ${{ secrets.DISCORD_TEST_GUILD_ID }}
DISCORD_TEST_CHANNEL_ID: ${{ secrets.DISCORD_TEST_CHANNEL_ID }}
SIGNAL_CLI_URL: ${{ secrets.SIGNAL_CLI_URL }}
SIGNAL_TEST_NUMBER: ${{ secrets.SIGNAL_TEST_NUMBER }}
IMESSAGE_BRIDGE_URL: ${{ secrets.IMESSAGE_BRIDGE_URL }}
IMESSAGE_TEST_HANDLE: ${{ secrets.IMESSAGE_TEST_HANDLE }}
WHATSAPP_TOKEN: ${{ secrets.WHATSAPP_TOKEN }}
WHATSAPP_PHONE_ID: ${{ secrets.WHATSAPP_PHONE_ID }}
WHATSAPP_TEST_CONTACT: ${{ secrets.WHATSAPP_TEST_CONTACT }}
TWILIO_ACCOUNT_SID: ${{ secrets.TWILIO_ACCOUNT_SID }}
TWILIO_AUTH_TOKEN: ${{ secrets.TWILIO_AUTH_TOKEN }}
TWILIO_FROM_NUMBER: ${{ secrets.TWILIO_FROM_NUMBER }}
TWILIO_TEST_TO_NUMBER: ${{ secrets.TWILIO_TEST_TO_NUMBER }}
X_API_KEY: ${{ secrets.X_API_KEY }}
X_API_SECRET: ${{ secrets.X_API_SECRET }}
X_ACCESS_TOKEN: ${{ secrets.X_ACCESS_TOKEN }}
X_ACCESS_SECRET: ${{ secrets.X_ACCESS_SECRET }}
X_TEST_DM_HANDLE: ${{ secrets.X_TEST_DM_HANDLE }}
NOTIFICATION_RELAY_URL: ${{ secrets.NOTIFICATION_RELAY_URL }}
NOTIFICATION_RELAY_TOKEN: ${{ secrets.NOTIFICATION_RELAY_TOKEN }}
TRAVEL_BOOKING_API_KEY: ${{ secrets.TRAVEL_BOOKING_API_KEY }}
SCENARIO_FILTER: ${{ inputs.scenario_filter }}
LIFEOPS_JUDGE_THRESHOLD: ${{ inputs.judge_threshold || '0.8' }}
BENCHMARK_ENFORCE_GATE: ${{ inputs.enforce_gate && '1' || '0' }}
BENCHMARK_REPORT_PATH: artifacts/benchmark-report.md
# Trajectory data and ephemeral state land under ~/.eliza per
# eliza-native ELIZA_STATE_DIR convention.
run: node packages/scripts/run-scenario-benchmark.mjs
- name: Upload benchmark report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: benchmark-report
path: |
artifacts/benchmark-report.md
artifacts/lifeops-scenario-report.json
if-no-files-found: warn
retention-days: 90
+118
View File
@@ -0,0 +1,118 @@
name: Browser Benchmark Lanes
# Deferred "Needs CI infra" lanes of #9476, tracked by #10333:
# - MiniWoB++ benchmark suite against a REAL Chromium engine (puppeteer-core)
# - ScreenSpot-Web-style grounding against a REAL Chromium engine
# - committed Mind2Web/WebArena-style external dataset fixtures through the
# plugin-browser BROWSER command router, including a REAL Chromium pass
#
# The `*.real.test.ts` lanes are excluded from the default `vitest run` and
# self-skip without a Chromium binary, so they only execute here after
# `bunx playwright install chromium` (mirroring scenario-pr.yml's
# `app-browser-core`). Gated like the other heavy real lanes — nightly +
# on-demand + on changes to the benchmark/executor — not on every PR.
on:
schedule:
# 10:00 UTC daily — after the external-API live lanes settle.
- cron: "0 10 * * *"
workflow_dispatch: {}
pull_request:
branches: [main]
paths:
- "plugins/plugin-browser/src/benchmark/**"
- "plugins/plugin-browser/vitest.real.config.ts"
- "plugins/plugin-browser/scripts/*benchmark*.mjs"
- "plugins/plugin-browser/scripts/capture-web-grounding-evidence.mjs"
- "plugins/plugin-browser/package.json"
- ".github/workflows/browser-real-bench.yml"
concurrency:
group: browser-real-bench-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
CI: "true"
NODE_VERSION: "24"
BUN_VERSION: "canary"
jobs:
browser-real-bench:
name: Browser benchmarks on real Chromium
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 30
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
# setup-bun-workspace instead of a hand-rolled `bun install
# --frozen-lockfile`: this repo tracks bun canary, and canary rebuilds
# reserialize bun.lock between builds, so a bare frozen install reds with
# "lockfile had changes, but lockfile is frozen" on schedule runs (e.g.
# run 28513244717). The composite handles the drift fallback and restores
# the committed lockfile; same invocation as scenario-pr.yml's
# app-browser-core lane this workflow mirrors.
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install --ignore-scripts --no-frozen-lockfile
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
# Real Chromium for the engine lane — the same pattern scenario-pr.yml's
# app-browser-core uses. resolveChromiumExecutablePath() picks it up via
# playwright-core, so no extra env wiring is needed.
- name: Install Chromium
run: bunx playwright install --with-deps chromium
- name: Run real Chromium benchmark lanes
run: bun run --cwd plugins/plugin-browser test:real-chromium
- name: Generate MiniWoB++ run-report artifacts (oracle + noop)
env:
BROWSER_EVIDENCE_DIR: ${{ runner.temp }}/evidence/10333-browser-real-chromium
run: |
mkdir -p "$BROWSER_EVIDENCE_DIR"
bun run --cwd plugins/plugin-browser bench:miniwob:chromium --policy oracle --seeds 3 --out "$BROWSER_EVIDENCE_DIR/miniwob-chromium-oracle-run.json"
bun run --cwd plugins/plugin-browser bench:miniwob:chromium --policy noop --seeds 2 --out "$BROWSER_EVIDENCE_DIR/miniwob-chromium-noop-run.json"
- name: Generate web-grounding artifacts
env:
OUT: ${{ runner.temp }}/evidence/10333-web-grounding
run: bun run --cwd plugins/plugin-browser bench:grounding:chromium
- name: Generate external-dataset artifacts
env:
BROWSER_EVIDENCE_DIR: ${{ runner.temp }}/evidence/10333-browser-external-dataset
run: |
mkdir -p "$BROWSER_EVIDENCE_DIR"
bun run --cwd plugins/plugin-browser bench:external --policy oracle --out "$BROWSER_EVIDENCE_DIR/external-dataset-oracle-run.json"
bun run --cwd plugins/plugin-browser bench:external --policy noop --out "$BROWSER_EVIDENCE_DIR/external-dataset-noop-run.json"
bun run --cwd plugins/plugin-browser bench:external --policy wrong --out "$BROWSER_EVIDENCE_DIR/external-dataset-wrong-run.json"
bun run --cwd plugins/plugin-browser bench:external:chromium --policy oracle --out "$BROWSER_EVIDENCE_DIR/external-dataset-chromium-oracle-run.json"
bun run --cwd plugins/plugin-browser bench:external:chromium --policy noop --out "$BROWSER_EVIDENCE_DIR/external-dataset-chromium-noop-run.json"
bun run --cwd plugins/plugin-browser bench:external:chromium --policy wrong --out "$BROWSER_EVIDENCE_DIR/external-dataset-chromium-wrong-run.json"
- name: Upload benchmark artifacts
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: browser-real-benchmark-runs
path: |
${{ runner.temp }}/evidence/10333-browser-real-chromium/*
${{ runner.temp }}/evidence/10333-web-grounding/*
${{ runner.temp }}/evidence/10333-browser-external-dataset/*.json
if-no-files-found: warn
+346
View File
@@ -0,0 +1,346 @@
name: Build Agent Image
# Single source of truth for the cloud-agent container image used by the
# cloud sandbox provisioner. Three trigger surfaces:
# - push to develop/main → :develop / :stable + :latest, plus :sha-<short>
# - GitHub release created → release tag (e.g. v2.0.0-alpha.176)
# - workflow_dispatch → manual reruns
# The previous release-only workflow (image.yaml) duplicated this entire
# pre-build sequence; it's been deleted in favor of this consolidated build.
on:
push:
branches: [develop, main]
paths:
- "packages/app-core/deploy/Dockerfile.ci"
- "plugins/plugin-sql/**"
- "plugins/plugin-elizacloud/**"
- "plugins/plugin-local-inference/**"
- "packages/agent/**"
- "packages/server/**"
- "packages/core/**"
- "packages/app/**"
- "packages/app-core/**"
- "packages/cloud/sdk/**"
- ".github/workflows/build-agent-image.yml"
- ".dockerignore"
release:
types: [created]
workflow_dispatch:
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}
# Default to least privilege. Override per-job where needed.
# cancel superseded in-flight runs for the same PR to free hosted-runner
# capacity. only cancels pull_request runs; push runs on protected branches
# (main/develop) are never cancelled mid-flight.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
permissions:
contents: read
jobs:
build-and-push:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 60
permissions:
contents: read
packages: write
attestations: write
id-token: write
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
sudo docker image prune -a -f || true
- name: Install jq
# Used by docker-ci-smoke's boot-KPI budget parser after the image is
# built. Keep it explicit so the real boot gate reports the committed
# budget instead of falling back to its baked-in default.
run: sudo apt-get install -y --no-install-recommends jq
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: "canary"
- name: Cache Bun install
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: ~/.bun/install/cache
key: bun-${{ runner.os }}-canary-${{ hashFiles('bun.lock') }}
restore-keys: |
bun-${{ runner.os }}-canary-
bun-${{ runner.os }}-
- name: Install dependencies
# Use `--no-frozen-lockfile` so a downstream lockfile drift in
# `develop` doesn't short-circuit the install before postinstall
# gets a chance to run (which is what was happening with the
# earlier `--frozen-lockfile` + fallback dance — the frozen
# install would fail, Bun's silent retry would no-op against
# the already-extracted virtual store, and postinstall would
# never run, leaving root-level deps like `uuid`, `drizzle-orm`,
# and the patched `tsup/dist/rollup.js` unavailable to the
# per-plugin build loop). The heavy submodule fetch is guarded
# by NODE_LLAMA_CPP_SKIP_DOWNLOAD.
env:
NODE_LLAMA_CPP_SKIP_DOWNLOAD: "true"
run: |
bun install --no-frozen-lockfile
- name: Cache Turbo build outputs
# The turbo build below is the single biggest pre-docker cost (~5-6
# min from cold). Nothing persisted turbo's local task cache between
# runs, so every push rebuilt every filtered package from scratch.
# Keying on the commit SHA (with a prefix restore-key) restores the
# most recent cache and saves an updated one each run, so a typical
# source commit only rebuilds the packages it actually touched.
# The key prefix is workflow-specific: entries here are produced
# under NODE_ENV=production, which turbo does not hash, so they must
# not be replayed into other workflows' builds.
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: .turbo/cache
key: turbo-agent-image-${{ runner.os }}-${{ github.sha }}
restore-keys: |
turbo-agent-image-${{ runner.os }}-
- name: Apply postinstall patches
# Belt-and-suspenders for the postinstall path: even with
# `bun install` (postinstall enabled), the silent fallback that
# kicks in when frozen-lockfile drifts doesn't always re-run the
# repo postinstall in CI. Explicitly run the patches the
# per-plugin builds depend on:
# - ensure-type-package-aliases: materializes @types/* under
# node_modules/@types so `tsc --build` resolves them.
env:
NODE_LLAMA_CPP_SKIP_DOWNLOAD: "true"
run: |
node packages/app-core/scripts/ensure-type-package-aliases.mjs
- name: Build Docker workspace artifacts
# Let Turbo derive the build order and reuse task cache instead of
# hand-maintaining the core/package/plugin sequence. The explicit
# filters include the app, the agent entrypoint, and every dynamically
# loaded plugin whose dist must exist before Dockerfile.ci relinks the
# local workspace packages into the image.
env:
NODE_ENV: production
NODE_LLAMA_CPP_SKIP_DOWNLOAD: "true"
run: |
node packages/scripts/run-turbo.mjs run build --concurrency=8 \
--filter=@elizaos/app \
--filter=@elizaos/agent \
--filter=@elizaos/plugin-sql \
--filter=@elizaos/plugin-video \
--filter=@elizaos/plugin-agent-skills \
--filter=@elizaos/plugin-pdf \
--filter=@elizaos/plugin-browser \
--filter=@elizaos/plugin-capacitor-bridge \
--filter=@elizaos/plugin-coding-tools \
--filter=@elizaos/plugin-native-filesystem \
--filter=@elizaos/plugin-shell \
--filter=@elizaos/plugin-commands \
--filter=@elizaos/plugin-computeruse \
--filter=@elizaos/plugin-discord \
--filter=@elizaos/plugin-edge-tts \
--filter=@elizaos/plugin-elizacloud \
--filter=@elizaos/plugin-imessage \
--filter=@elizaos/plugin-local-inference \
--filter=@elizaos/plugin-mcp \
--filter=@elizaos/plugin-signal \
--filter=@elizaos/plugin-streaming \
--filter=@elizaos/plugin-telegram \
--filter=@elizaos/plugin-whatsapp \
--filter=@elizaos/plugin-wallet \
--filter=@elizaos/plugin-workflow \
--filter=@elizaos/plugin-x402
- name: Prune stale Turbo cache entries
# actions/cache restores + re-saves the whole cache dir, so without a
# prune the tarball grows by every commit's changed-package outputs
# forever. tar preserves mtimes, so age-based deletion bounds it:
# entries untouched for two weeks are long since superseded.
run: find .turbo/cache -type f -mtime +14 -delete 2>/dev/null || true
- name: Normalize plugin dist for Node ESM
# MUST run AFTER the Turbo build above. Some package builds still emit
# extensionless or directory specifiers (`./actions/index`,
# `./approvals`) in dist. The container loads those built `.js` files
# via import(), and neither tsx nor plain `node` resolves them. Apply
# the same rewrite the agent's own build uses to EVERY built plugin
# dist, last, so the dist that actually ships is Node-ESM-resolvable.
run: |
shopt -s nullglob
for dist in plugins/*/dist; do
pkg="$(dirname "$dist")"
node packages/scripts/rewrite-dist-relative-imports-node-esm.mjs "$pkg" || true
done
- name: Prepare CI dockerignore
# The root .dockerignore is tuned for `bun install` runs (excludes
# everything heavy). The Docker build needs a different policy:
# keep root node_modules (Dockerfile.ci's pruner relink scripts
# consume them) but strip per-package node_modules and the other
# bulky CI-only inputs. Use the canonical CI dockerignore from
# packages/app-core/deploy/.dockerignore.ci, mirroring what
# docker-ci-smoke.sh does before its `docker build`.
run: |
cp packages/app-core/deploy/.dockerignore.ci .dockerignore
- uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c
with:
driver-opts: |
env.BUILDKIT_STEP_LOG_MAX_SIZE=10485760
env.BUILDKIT_STEP_LOG_MAX_SPEED=10485760
# github.repository preserves the org's casing (elizaOS/eliza), but Docker
# tags must be all-lowercase. metadata-action lowercases for the push tags,
# but the raw :ci-boot-verify tag below needs an explicit lowercased image
# name or `docker build`/`docker push` fails with "repository name must be
# lowercase".
- id: image
run: echo "name=${REGISTRY}/${IMAGE_NAME,,}" >> "$GITHUB_OUTPUT"
- id: meta
uses: docker/metadata-action@80c7e94dd9b9319bd5eb7a0e0fe9291e23a2a2e9
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
# develop push → :develop + :sha-<short>
# main push → :stable + :latest + :sha-<short>
# release → the git tag verbatim (e.g. v2.0.0-alpha.176)
# Keeping :latest on main avoids breaking the four code sites that
# still hardcode it (containers-env fallback, sandbox-provider
# fallback, two test fixtures) until they migrate to :stable.
tags: |
type=raw,value=develop,enable=${{ github.ref == 'refs/heads/develop' }}
type=raw,value=stable,enable=${{ github.ref == 'refs/heads/main' }}
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }}
type=ref,event=tag
type=sha,prefix=sha-,format=short
# Build the image into the local Docker daemon first (load, do NOT push).
# The boot-verification step below runs the REAL agent entrypoint against
# this exact image, and the push step ships these same loaded bytes by
# retagging the local image. This is the gate that stops a container which
# crashes on startup (missing runtime deps, broken entrypoint) from
# reaching ghcr. `provenance: false` keeps the loaded image a single-
# platform image so the retag-and-push below publishes the same content
# that was verified (no second build, no "latest yt-dlp" drift between the
# verified and published image).
- id: build-local
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a
with:
context: .
file: packages/app-core/deploy/Dockerfile.ci
push: false
load: true
provenance: false
tags: ${{ steps.image.outputs.name }}:ci-boot-verify
labels: ${{ steps.meta.outputs.labels }}
platforms: linux/amd64
# Registry-backed build cache (checked first), with the old GHA
# cache kept as a read-only fallback. The GHA backend shares one
# repo-wide 10GB budget with every other workflow; this image's
# mode=max cache is multi-GB per run, so it was evicted before the
# next develop push — recent runs show ZERO cached layers while
# still spending 4+ min per run exporting a cache nothing ever
# read (which is also why cache-to no longer writes to gha). The
# ghcr :buildcache ref persists indefinitely and dedupes by blob
# digest, so stable layers (apt/tailscale/yt-dlp/tsx base, and the
# manifest-keyed runtime-deps npm closure) hit reliably and the
# per-run cache export only uploads blobs ghcr doesn't already
# have. image-manifest+oci-mediatypes make the cache manifest a
# ghcr-compatible OCI artifact.
cache-from: |
type=registry,ref=${{ steps.image.outputs.name }}:buildcache
type=gha
cache-to: type=registry,ref=${{ steps.image.outputs.name }}:buildcache,mode=max,image-manifest=true,oci-mediatypes=true
- name: Verify image boots (real agent entrypoint)
# Runs the production entrypoint (node --import
# /opt/tsx/node_modules/tsx/dist/loader.mjs packages/agent/dist/bin.js
# start) inside the just-built image and fails RED if the agent crashes
# on startup, exits non-zero, logs a known crash signature (Cannot find
# package/module, Failed to start, crashed during init, ...), never
# serves /api/health within the timeout, or crashes during the
# post-health stability window. A failure here blocks the push below, so
# a non-bootable image is never published.
env:
DOCKER_IMAGE: ${{ steps.image.outputs.name }}:ci-boot-verify
BOOT_VERIFY_ONLY: "true"
SMOKE_TIMEOUT_SEC: "180"
BOOT_STABLE_WINDOW_SEC: "15"
SMOKE_PORT: "32138"
CONTAINER_PORT: "42138"
# Enforce the boot-KPI cold-readyMs budget (#8812 item 5). The budget
# (boot.coldReadyMs, 25000ms) carries ~5x headroom over the ~5s real
# cold boot, and emit_boot_kpi downgrades a breach to a warning when the
# runner is heavily contended, so this only fails on a genuine multi-x
# boot-time regression — not on transient runner load.
BOOT_KPI_ENFORCE: "1"
run: bash packages/app-core/scripts/docker-ci-smoke.sh --boot-verify-only
# Push the EXACT verified image. This runs only if the boot verification
# above succeeded (steps run in order; a failed step fails the job). We
# retag the locally-loaded :ci-boot-verify image to each computed tag and
# `docker push` it, so the published bytes are byte-for-byte the image
# that just booted cleanly. No rebuild means no chance of shipping a
# different image than the one that was verified.
- id: push
env:
LOCAL_IMAGE: ${{ steps.image.outputs.name }}:ci-boot-verify
TAGS: ${{ steps.meta.outputs.tags }}
run: |
set -euo pipefail
first_tag=""
while IFS= read -r tag; do
[ -z "$tag" ] && continue
[ -z "$first_tag" ] && first_tag="$tag"
echo "Tagging $LOCAL_IMAGE as $tag and pushing"
docker tag "$LOCAL_IMAGE" "$tag"
docker push "$tag"
done <<< "$TAGS"
# Resolve the published digest from a pushed tag (its RepoDigests is
# populated by the push above) for the attestation step.
digest=""
if [ -n "$first_tag" ]; then
digest="$(docker inspect --format '{{ index .RepoDigests 0 }}' "$first_tag" 2>/dev/null | sed 's/.*@//' || true)"
fi
echo "digest=$digest" >> "$GITHUB_OUTPUT"
- name: Generate artifact attestation
if: ${{ steps.push.outputs.digest != '' }}
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373
with:
subject-name: ${{ steps.image.outputs.name }}
subject-digest: ${{ steps.push.outputs.digest }}
push-to-registry: true
- name: Make Docker image public
# Idempotent — patching to public when already public is a no-op.
# Previously lived in image.yaml; folded in here on consolidation.
run: |
curl \
-X PATCH \
-H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/user/packages/container/${{ env.IMAGE_NAME }}/visibility \
-d '{"visibility":"public"}'
+250
View File
@@ -0,0 +1,250 @@
name: Build Android
on:
workflow_dispatch:
inputs:
upload_to_release:
description: "Upload APK/AAB to latest GitHub release"
required: false
type: boolean
default: false
concurrency:
group: build-android-${{ github.ref }}
cancel-in-progress: true
env:
APP_DIR: packages/app
ANDROID_DIR: packages/app-core/platforms/android
NODE_VERSION: "24.15.0"
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
build-android:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 30
permissions:
actions: read
contents: write
steps:
- name: Checkout (with the llama.cpp fork submodule)
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
# The elizavoice JNI wrapper compiles against the omnivoice headers
# inside plugins/plugin-local-inference/native/llama.cpp — the staged
# fused lib exports the fused-voice ABI, so the headers must be
# checked out or build:android fails at [elizavoice-jni] (#15540).
submodules: recursive
show-progress: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: canary
- name: Setup JDK 21
uses: actions/setup-java@ad2b38190b15e4d6bdf0c97fb4fca8412226d287
with:
distribution: temurin
java-version: "21"
- name: Setup Android SDK
uses: android-actions/setup-android@40fd30fb8d7440372e1316f5d1809ec01dcd3699
- name: Cache Gradle
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: |
~/.gradle/caches
~/.gradle/wrapper
key: gradle-${{ runner.os }}-${{ hashFiles('packages/app-core/platforms/android/**/*.gradle*', 'packages/app-core/platforms/android/gradle/wrapper/gradle-wrapper.properties', 'packages/app/capacitor.config.ts') }}
restore-keys: gradle-${{ runner.os }}-
- name: Install dependencies
run: bun install
# Gradle's copyForkLlamaLib hard-requires the fused inference lib for
# arm64-v8a (libelizainference.so + its backend .so siblings) and fails
# the whole APK build without it (#15540). CI has no ~/.eliza staging
# and no NDK cross-compile here — the fused libs are built by the
# build-llama-ffi-android workflow. Native artifacts are valid only for
# the exact producer inputs in this checkout; mixing a newer or older
# llama pin with these JNI headers can link successfully and fail later
# on-device.
- name: Stage fused inference libs (from build-llama-ffi-android)
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
producer_inputs=(
"plugins/plugin-local-inference/native/llama.cpp"
".github/workflows/build-llama-ffi-android.yml"
"packages/app-core/scripts/aosp/compile-libllama.mjs"
"packages/app-core/scripts/aosp/compile-libllama-paths.mjs"
)
run_id=""
while IFS=$'\t' read -r candidate source_sha; do
compatible=true
for path in "${producer_inputs[@]}"; do
local_sha=$(git rev-parse "HEAD:$path")
source_sha_for_path=$(gh api "repos/${{ github.repository }}/contents/$path?ref=$source_sha" \
--jq '.sha // empty' 2>/dev/null || true)
if [ "$source_sha_for_path" != "$local_sha" ]; then
compatible=false
break
fi
done
if [ "$compatible" = true ]; then
run_id="$candidate"
break
fi
done < <(gh api "repos/${{ github.repository }}/actions/workflows/build-llama-ffi-android.yml/runs?status=success&per_page=20" \
--jq '.workflow_runs[] | [.id, .head_sha] | @tsv')
if [ -z "$run_id" ]; then
echo "::error::no successful build-llama-ffi-android run matches this checkout's native producer inputs — build that workflow first" >&2
exit 1
fi
echo "staging fused libs from input-compatible build-llama-ffi-android run $run_id"
stage="$RUNNER_TEMP/elizainference"
mkdir -p "$stage/arm64" "$stage/x86_64"
gh run download "$run_id" -R "${{ github.repository }}" \
-n libelizainference-android-arm64-vulkan-fused -D "$stage/arm64"
# x86_64 is optional: gradle skips absent non-arm64 ABIs.
gh run download "$run_id" -R "${{ github.repository }}" \
-n libelizainference-android-x86_64-cpu-fused -D "$stage/x86_64" || true
arm64_so=$(find "$stage/arm64" -name libelizainference.so | head -1)
if [ -z "$arm64_so" ]; then
echo "::error::arm64 artifact downloaded but contains no libelizainference.so" >&2
find "$stage/arm64" -type f | head -20 >&2
exit 1
fi
echo "ELIZA_MTP_ANDROID_LIBDIR=$(dirname "$arm64_so")" >> "$GITHUB_ENV"
x86_so=$(find "$stage/x86_64" -name libelizainference.so 2>/dev/null | head -1 || true)
if [ -n "$x86_so" ]; then
echo "ELIZA_MTP_ANDROID_LIBDIR_X86_64=$(dirname "$x86_so")" >> "$GITHUB_ENV"
fi
echo "arm64 libdir contents:"
ls -la "$(dirname "$arm64_so")"
- name: Prepare Android platform
working-directory: ${{ env.APP_DIR }}
run: bun run build:android
- name: Run Android developer-install preflight
working-directory: ${{ env.APP_DIR }}
run: bun run preflight:android:sideload
- name: Override Gradle JDK path for CI
working-directory: ${{ env.ANDROID_DIR }}
run: |
# Remove the hardcoded local JDK path so Gradle uses JAVA_HOME from setup-java
sed -i '/org.gradle.java.home=/d' gradle.properties
- name: Determine version
id: version
run: |
if [[ "${{ github.event_name }}" == "release" ]]; then
VERSION="${{ github.event.release.tag_name }}"
VERSION="${VERSION#v}"
else
VERSION="0.0.0-dev"
fi
echo "version=$VERSION" >> $GITHUB_OUTPUT
# Compute versionCode from semver (major*10000 + minor*100 + patch)
IFS='.-' read -r major minor patch _ <<< "$VERSION"
VERSION_CODE=$(( ${major:-0} * 10000 + ${minor:-0} * 100 + ${patch:-0} ))
echo "version_code=$VERSION_CODE" >> $GITHUB_OUTPUT
- name: Build debug APK
if: github.event_name == 'workflow_dispatch'
working-directory: ${{ env.ANDROID_DIR }}
env:
ELIZAOS_VERSION_NAME: ${{ steps.version.outputs.version }}
ELIZAOS_VERSION_CODE: ${{ steps.version.outputs.version_code }}
run: ./gradlew assembleDebug --no-daemon
- name: Build release APK & AAB
if: github.event_name == 'release'
working-directory: ${{ env.ANDROID_DIR }}
env:
ELIZAOS_VERSION_NAME: ${{ steps.version.outputs.version }}
ELIZAOS_VERSION_CODE: ${{ steps.version.outputs.version_code }}
run: |
./gradlew assembleRelease --no-daemon
./gradlew bundleRelease --no-daemon
- name: Sign release APK
if: github.event_name == 'release' && env.ANDROID_KEYSTORE_BASE64 != ''
env:
ANDROID_KEYSTORE_BASE64: ${{ secrets.ANDROID_KEYSTORE_BASE64 }}
ANDROID_KEYSTORE_PASSWORD: ${{ secrets.ANDROID_KEYSTORE_PASSWORD }}
ANDROID_KEY_ALIAS: ${{ secrets.ANDROID_KEY_ALIAS }}
ANDROID_KEY_PASSWORD: ${{ secrets.ANDROID_KEY_PASSWORD }}
run: |
echo "$ANDROID_KEYSTORE_BASE64" | base64 -d > /tmp/keystore.jks
UNSIGNED_APK="${{ env.ANDROID_DIR }}/app/build/outputs/apk/release/app-release-unsigned.apk"
SIGNED_APK="${{ env.ANDROID_DIR }}/app/build/outputs/apk/release/eliza-${{ steps.version.outputs.version }}.apk"
${ANDROID_HOME}/build-tools/35.0.0/apksigner sign \
--ks /tmp/keystore.jks \
--ks-pass "pass:${ANDROID_KEYSTORE_PASSWORD}" \
--ks-key-alias "${ANDROID_KEY_ALIAS}" \
--key-pass "pass:${ANDROID_KEY_PASSWORD}" \
--out "$SIGNED_APK" \
"$UNSIGNED_APK"
rm /tmp/keystore.jks
- name: Generate Android checksums
if: github.event_name == 'release'
run: |
find "${{ env.ANDROID_DIR }}/app/build/outputs" \
\( -name "*.apk" -o -name "*.aab" \) -type f -print0 \
| sort -z \
| xargs -0 sha256sum > "${{ env.ANDROID_DIR }}/app/build/outputs/SHA256SUMS.txt"
cat "${{ env.ANDROID_DIR }}/app/build/outputs/SHA256SUMS.txt"
- name: Upload debug APK artifact
if: github.event_name == 'workflow_dispatch'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: eliza-android-debug
path: ${{ env.ANDROID_DIR }}/app/build/outputs/apk/debug/app-debug.apk
if-no-files-found: error
retention-days: 7
- name: Upload release artifacts
if: github.event_name == 'release'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: eliza-android-release
path: |
${{ env.ANDROID_DIR }}/app/build/outputs/apk/release/*.apk
${{ env.ANDROID_DIR }}/app/build/outputs/bundle/release/*.aab
${{ env.ANDROID_DIR }}/app/build/outputs/SHA256SUMS.txt
if-no-files-found: error
retention-days: 30
- name: Upload to GitHub Release
if: github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && inputs.upload_to_release)
uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b
with:
tag_name: ${{ github.event_name == 'release' && github.event.release.tag_name || '' }}
files: |
${{ env.ANDROID_DIR }}/app/build/outputs/apk/release/*.apk
${{ env.ANDROID_DIR }}/app/build/outputs/bundle/release/*.aab
${{ env.ANDROID_DIR }}/app/build/outputs/apk/debug/app-debug.apk
${{ env.ANDROID_DIR }}/app/build/outputs/SHA256SUMS.txt
fail_on_unmatched_files: false
+164
View File
@@ -0,0 +1,164 @@
name: Build elizaOS Debian Package
on:
workflow_call:
workflow_dispatch:
release:
types: [created]
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
build-deb:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 30
permissions:
# contents: write covers the existing softprops/action-gh-release
# release upload + checkout.
# id-token + attestations are required by
# actions/attest-build-provenance@v4 to mint a Sigstore-signed
# SLSA build provenance for the .deb via GitHub OIDC.
contents: write
id-token: write
attestations: write
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Install Debian build tools
run: |
sudo apt-get update -qq
sudo apt-get install -y --no-install-recommends \
dpkg-dev \
devscripts \
debhelper \
dh-make \
fakeroot \
lintian
- name: Set up Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: "canary"
- name: Install dependencies
run: bun install --frozen-lockfile || bun install --no-frozen-lockfile
- name: Build
run: bun run build
- name: Build .deb package
id: deb
run: |
set -euo pipefail
if [ ! -d packages/app-core/packaging/debian ]; then
echo "ERROR: packages/app-core/packaging/debian is missing — debian packaging dir is required for the .deb build. Repo state is bad."
exit 1
fi
# dpkg-buildpackage expects the source root to have a debian/ subdir
# alongside the source it needs to build. The repo keeps the
# packaging definitions under packages/app-core/packaging/debian to
# avoid clutter; link them into the app-core source root for the
# build, then unlink so a follow-up run is idempotent.
cd packages/app-core
if [ ! -L debian ]; then
ln -sfn packaging/debian debian
fi
dpkg-buildpackage -us -uc -b -d
rm -f debian
- name: Find and checksum .deb
id: find_deb
run: |
set -euo pipefail
DEB=$(find . -maxdepth 3 -name "*.deb" | head -1)
if [ -z "$DEB" ]; then
echo "ERROR: no .deb file produced after dpkg-buildpackage"
exit 1
fi
sha256sum "$DEB" > "${DEB}.sha256"
DEB_SHA256=$(cut -d' ' -f1 "${DEB}.sha256")
DEB_SIZE=$(stat --format="%s" "$DEB")
echo "deb_path=$DEB" >> "$GITHUB_OUTPUT"
echo "deb_sha256=$DEB_SHA256" >> "$GITHUB_OUTPUT"
echo "deb_size=$DEB_SIZE" >> "$GITHUB_OUTPUT"
echo "Found .deb: $DEB ($DEB_SIZE bytes)"
- name: Lint .deb with lintian
# lintian is installed in the "Install Debian build tools" step above.
# Findings are surfaced as GitHub annotations but do NOT fail the build —
# this is a visibility step. Flip to --fail-on-error once noise is acceptable.
if: steps.find_deb.outputs.deb_path != ''
run: |
set +e
lintian -i --pedantic --no-tag-display-limit "${{ steps.find_deb.outputs.deb_path }}" > lintian.log 2>&1
LINTIAN_EXIT=$?
set -e
while IFS= read -r line; do
case "$line" in
E:*) echo "::error::lintian: $line" ;;
W:*) echo "::warning::lintian: $line" ;;
I:*) echo "::notice::lintian: $line" ;;
P:*) echo "::notice::lintian: $line" ;;
esac
done < lintian.log
echo "lintian exit code: $LINTIAN_EXIT (annotations only; not failing build)"
- name: Upload lintian log
if: always() && steps.find_deb.outputs.deb_path != ''
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: elizaos-debian-lintian-log
path: lintian.log
if-no-files-found: ignore
retention-days: 30
- name: Attest SLSA build provenance for .deb
# Mints a Sigstore-signed in-toto/SLSA provenance attestation
# tied to this repo + workflow + commit, via GitHub OIDC.
# No long-lived secrets. Users verify with:
# gh attestation verify <deb-file> --owner elizaOS
if: steps.find_deb.outputs.deb_path != ''
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373
with:
subject-path: ${{ steps.find_deb.outputs.deb_path }}
- name: Upload .deb artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: elizaos-debian-package
path: |
**/*.deb
**/*.deb.sha256
if-no-files-found: error
retention-days: 30
- name: Upload to GitHub Release
if: (github.event_name == 'release') && steps.find_deb.outputs.deb_path != ''
uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b
with:
files: |
**/*.deb
**/*.deb.sha256
fail_on_unmatched_files: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Write .deb metadata to summary
run: |
{
echo "## elizaOS Debian Package"
echo "| Field | Value |"
echo "| ----- | ----- |"
if [ -n "${{ steps.find_deb.outputs.deb_path }}" ]; then
echo "| File | \`${{ steps.find_deb.outputs.deb_path }}\` |"
echo "| SHA-256 | \`${{ steps.find_deb.outputs.deb_sha256 }}\` |"
echo "| Size | ${{ steps.find_deb.outputs.deb_size }} bytes |"
else
echo "| Status | not produced (packaging/debian may not be present yet) |"
fi
} >> "$GITHUB_STEP_SUMMARY"
@@ -0,0 +1,354 @@
name: Build Example App Images
# Builds + pushes standalone example-app images as a smoke/publish lane. The
# #9300 real-staging showcase loop deploys these apps from source through the
# normal app deploy builder (`POST /api/v1/apps/:id/deploy` with repo/ref/
# Dockerfile), not by pinning these tags operator-side.
#
# Why we bundle on the runner instead of building ./Dockerfile from scratch:
# server.ts imports `@elizaos/cloud-sdk` (workspace:*). That only resolves
# inside the monorepo, and the SDK's dist/ is gitignored. A standalone docker
# build of ./Dockerfile would crash at boot ("Cannot find package
# @elizaos/cloud-sdk"). Instead we run `bun build server.ts` on the runner —
# where the workspace IS resolvable — which inlines the SDK into a single
# self-contained server.js (verified: 0 residual @elizaos/cloud-sdk imports),
# then ship that with Dockerfile.bundle from a tiny context. Same pre-build
# sequence build-agent-image.yml uses to make the cloud-sdk dist available.
on:
push:
branches: [develop]
paths:
- "packages/examples/cloud/edad/**"
- "packages/examples/cloud/clone-ur-crush/**"
- "packages/cloud/sdk/**"
- ".github/workflows/build-example-app-images.yml"
workflow_dispatch:
# One in-flight build per ref; a newer push cancels the stale one (avoids the
# queue pile-up the agent-image build learned the hard way).
concurrency:
group: build-example-edad-${{ github.ref }}
cancel-in-progress: true
env:
REGISTRY: ghcr.io
IMAGE_NAME: elizaos/example-edad
# Least privilege by default; the build job opts into packages: write.
permissions:
contents: read
jobs:
build-edad:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 30
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: "canary"
- name: Cache Bun install
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: ~/.bun/install/cache
key: bun-${{ runner.os }}-canary-${{ hashFiles('bun.lock') }}
restore-keys: |
bun-${{ runner.os }}-canary-
bun-${{ runner.os }}-
- name: Install dependencies
# Creates the @elizaos/cloud-sdk workspace symlink edad resolves through.
# --no-frozen-lockfile mirrors build-agent-image.yml: a downstream
# lockfile drift on develop shouldn't short-circuit the install.
env:
NODE_LLAMA_CPP_SKIP_DOWNLOAD: "true"
run: bun install --no-frozen-lockfile
- name: Build @elizaos/cloud-sdk
# The SDK's `exports.import` points at ./dist/index.js, which is
# gitignored (absent on a cold checkout). Without this build, the
# `bun build` below fails to resolve "@elizaos/cloud-sdk" even with the
# workspace symlink in place. build-agent-image.yml builds this same
# package for the same reason.
env:
NODE_LLAMA_CPP_SKIP_DOWNLOAD: "true"
run: |
cd packages/cloud/sdk
bun install --no-frozen-lockfile --ignore-scripts
bun run build
- name: Bundle the EDAD server (inline the workspace SDK)
# Produces a single self-contained server.js with @elizaos/cloud-sdk
# inlined — no workspace dep, no bun install needed in the image. Output
# goes to ./dist (already gitignored, same dir edad's `build` script
# uses); dist/ (server.js + public/) is the tiny docker build context.
run: |
set -euo pipefail
cd packages/examples/cloud/edad
node ../../../../packages/scripts/rm-path-recursive.mjs ./dist
bun build server.ts --outdir=./dist --target=bun --format=esm
cp -r public ./dist/public
test -f ./dist/server.js
# Guard: the bundle must NOT still reference the workspace dep, else
# the image would crash at boot.
if grep -q "@elizaos/cloud-sdk" ./dist/server.js; then
echo "::error::server.js still references @elizaos/cloud-sdk — bundle did not inline the workspace dep" >&2
exit 1
fi
- name: Smoke-test the image (gate the GHCR push on a working container)
# Build + run the exact bundle image and assert it actually serves before
# we publish it — so the showcase deploy never pulls a broken :showcase
# tag. Mirrors the local verification in DEPLOY_AND_VALIDATE.md.
run: |
set -euo pipefail
assert_asset() {
local asset_path="$1"
local expected_type="$2"
local min_bytes="$3"
local headers
local body
headers="$(mktemp)"
body="$(mktemp)"
curl -fsS -D "$headers" -o "$body" "http://127.0.0.1:3000/${asset_path}"
if ! grep -qi "content-type: ${expected_type}" "$headers"; then
echo "::error::${asset_path} content-type was not ${expected_type}" >&2
cat "$headers" >&2
exit 1
fi
local bytes
bytes="$(wc -c < "$body" | tr -d ' ')"
if [ "$bytes" -lt "$min_bytes" ]; then
echo "::error::${asset_path} was too small (${bytes} bytes)" >&2
exit 1
fi
}
docker build --load -t edad-smoke:ci \
-f packages/examples/cloud/edad/Dockerfile.bundle \
packages/examples/cloud/edad/dist
docker rm -f edad-smoke >/dev/null 2>&1 || true
trap 'docker rm -f edad-smoke >/dev/null 2>&1 || true' EXIT
docker run -d --name edad-smoke -p 3000:3000 edad-smoke:ci
ok=""
for _ in $(seq 1 30); do
if curl -fsS http://127.0.0.1:3000/health >/dev/null 2>&1; then ok=1; break; fi
sleep 1
done
if [ -z "$ok" ]; then
echo "::error::EDAD image never became healthy" >&2
docker logs edad-smoke || true
exit 1
fi
test "$(curl -fsS http://127.0.0.1:3000/health)" = "ok"
curl -fsS http://127.0.0.1:3000/ | grep -q "eDad" \
|| { echo "::error::EDAD did not serve its UI" >&2; exit 1; }
curl -fsS http://127.0.0.1:3000/ | grep -q 'content="http://127.0.0.1:3000/og-image.png"' \
|| { echo "::error::EDAD did not emit an absolute social image URL" >&2; exit 1; }
curl -fsS http://127.0.0.1:3000/api/config | grep -q "cloud_url" \
|| { echo "::error::EDAD /api/config is broken" >&2; exit 1; }
assert_asset dad-portrait.svg image/svg+xml 4000
assert_asset dad-avatar.svg image/svg+xml 2000
assert_asset you-avatar.svg image/svg+xml 500
assert_asset favicon.svg image/svg+xml 500
assert_asset favicon.png image/png 1000
assert_asset apple-touch-icon.png image/png 4000
assert_asset og-image.png image/png 100000
docker rm -f edad-smoke >/dev/null 2>&1 || true
trap - EXIT
echo "EDAD image smoke test passed: healthy + serves UI + /api/config + generated assets."
- uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c
- id: meta
uses: docker/metadata-action@80c7e94dd9b9319bd5eb7a0e0fe9291e23a2a2e9
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
# :showcase → the moving tag the showcase spec deploys
# :sha-<short> → immutable per-commit tag (mirrors build-agent-image.yml)
tags: |
type=raw,value=showcase
type=sha,prefix=sha-,format=short
- id: build
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a
with:
context: packages/examples/cloud/edad/dist
file: packages/examples/cloud/edad/Dockerfile.bundle
push: true
provenance: false
platforms: linux/amd64
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Make image public
# The deploy puller pulls anonymously; keep the package public.
# Idempotent — patching to public when already public is a no-op.
run: |
curl --fail-with-body --show-error --silent \
-X PATCH \
-H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/orgs/elizaOS/packages/container/example-edad/visibility \
-d '{"visibility":"public"}' || \
echo "::warning::Could not set package visibility to public — set it once manually in the GHCR package settings"
build-clone-ur-crush:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 30
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: "canary"
- name: Cache Bun install
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: ~/.bun/install/cache
key: bun-${{ runner.os }}-canary-${{ hashFiles('bun.lock') }}
restore-keys: |
bun-${{ runner.os }}-canary-
bun-${{ runner.os }}-
- name: Install dependencies
env:
NODE_LLAMA_CPP_SKIP_DOWNLOAD: "true"
run: bun install --no-frozen-lockfile
- name: Build clone-ur-crush (Next.js standalone bundle)
# next.config sets output:"standalone"; with outputFileTracingRoot at the
# monorepo root the server lands at the repo-relative path inside the
# bundle. Next does NOT copy static/public into standalone, so we place
# them next to server.js where the standalone runtime serves them.
#
# CRITICAL (#9300): build.mjs builds under a temp distDir
# (.next-build-<pid>) then renames it to .next, but the standalone server
# BAKES that original distDir name into required-server-files.json and
# serves /_next/static from <bakedDistDir>/static. Copying static to the
# literal .next/static (the renamed dir) puts it at a path the runtime
# never reads, so every CSS/JS chunk 404s. Copy static to the BAKED dist
# dir instead.
env:
NODE_LLAMA_CPP_SKIP_DOWNLOAD: "true"
run: |
set -euo pipefail
cd packages/examples/cloud/clone-ur-crush
node ../../../../packages/scripts/rm-path-recursive.mjs .next
bun run build
app=.next/standalone/packages/examples/cloud/clone-ur-crush
test -f "$app/server.js"
# The standalone's baked distDir = the dir holding required-server-files.json.
rsf="$(find "$app" -maxdepth 2 -name required-server-files.json | head -1)"
test -n "$rsf"
baked="$(basename "$(dirname "$rsf")")"
echo "standalone baked distDir: $baked"
mkdir -p "$app/$baked/static"
cp -r .next/static/. "$app/$baked/static/"
cp -r public "$app/public"
- name: Smoke-test the image (gate the GHCR push on a working container)
# Build + run the exact standalone image and assert it actually serves its
# UI before we publish it, so the showcase deploy never pulls a broken
# :showcase tag. Mirrors the EDAD job's gate above (DoD: failures surface
# loudly; the showcase deploy can never pull a broken tag).
run: |
set -euo pipefail
docker build --load -t cuc-smoke:ci \
-f packages/examples/cloud/clone-ur-crush/Dockerfile.bundle \
packages/examples/cloud/clone-ur-crush/.next/standalone
docker rm -f cuc-smoke >/dev/null 2>&1 || true
trap 'docker rm -f cuc-smoke >/dev/null 2>&1 || true' EXIT
docker run -d --name cuc-smoke -p 3000:3000 cuc-smoke:ci
ok=""
for _ in $(seq 1 40); do
if curl -fsS http://127.0.0.1:3000/ >/dev/null 2>&1; then ok=1; break; fi
sleep 1
done
if [ -z "$ok" ]; then
echo "::error::Clone Ur Crush image never started serving" >&2
docker logs cuc-smoke || true
exit 1
fi
home="$(curl -fsS http://127.0.0.1:3000/)"
echo "$home" | grep -q "Clone Your Crush" \
|| { echo "::error::Clone Ur Crush did not serve its UI" >&2; docker logs cuc-smoke || true; exit 1; }
# Assert a real /_next/static chunk serves 200 — this is what catches the
# distDir/static path mismatch (a broken bundle serves HTML but 404s
# every CSS/JS asset). Pull a static URL out of the rendered HTML.
asset="$(printf '%s' "$home" | grep -oE '/_next/static/[^"]+\.(js|css)' | head -1)"
test -n "$asset" || { echo "::error::no /_next/static asset referenced in HTML" >&2; exit 1; }
code="$(curl -s -o /dev/null -w '%{http_code}' "http://127.0.0.1:3000${asset}")"
test "$code" = "200" \
|| { echo "::error::static asset ${asset} returned ${code} (expected 200) — broken /_next/static path" >&2; docker logs cuc-smoke || true; exit 1; }
# OG card must exist (was a 404 before #9300 fix).
og="$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:3000/og-image.png)"
test "$og" = "200" || { echo "::error::og-image.png returned ${og} (expected 200)" >&2; exit 1; }
docker rm -f cuc-smoke >/dev/null 2>&1 || true
trap - EXIT
echo "Clone Ur Crush image smoke test passed: serves UI + /_next/static (${asset}) + og-image.png."
- uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c
- id: meta
uses: docker/metadata-action@80c7e94dd9b9319bd5eb7a0e0fe9291e23a2a2e9
with:
images: ${{ env.REGISTRY }}/elizaos/example-clone-ur-crush
tags: |
type=raw,value=showcase
type=sha,prefix=sha-,format=short
- id: build
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a
with:
context: packages/examples/cloud/clone-ur-crush/.next/standalone
file: packages/examples/cloud/clone-ur-crush/Dockerfile.bundle
push: true
provenance: false
platforms: linux/amd64
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Make image public
run: |
curl --fail-with-body --show-error --silent \
-X PATCH \
-H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
-H "Accept: application/vnd.github.v3+json" \
https://api.github.com/orgs/elizaOS/packages/container/example-clone-ur-crush/visibility \
-d '{"visibility":"public"}' || \
echo "::warning::Could not set package visibility to public — set it once manually in the GHCR package settings"
+224
View File
@@ -0,0 +1,224 @@
name: Build iOS
on:
workflow_dispatch:
inputs:
upload_to_release:
description: "Upload IPA to latest GitHub release"
required: false
type: boolean
default: false
concurrency:
group: build-ios-${{ github.ref }}
cancel-in-progress: true
env:
APP_DIR: packages/app
NODE_VERSION: "24.15.0"
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
build-ios:
runs-on: macos-15
timeout-minutes: 45
permissions:
contents: write
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: canary
- name: Select latest Xcode
run: |
XCODE_PATH=$(ls -d /Applications/Xcode*.app 2>/dev/null | sort -V | tail -1)
echo "Using Xcode at: $XCODE_PATH"
sudo xcode-select -s "$XCODE_PATH"
xcodebuild -version
- name: Cache CocoaPods
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: ${{ env.APP_DIR }}/ios/App/Pods
key: pods-${{ runner.os }}-${{ hashFiles('packages/app-core/platforms/ios/App/Podfile', 'packages/app/capacitor.config.ts', 'packages/app/package.json') }}
restore-keys: pods-${{ runner.os }}-
- name: Install dependencies
run: bun install
- name: Build web assets
working-directory: ${{ env.APP_DIR }}
run: bun run build
- name: Build Capacitor plugins
working-directory: ${{ env.APP_DIR }}
run: bun run plugin:build
- name: Sync Capacitor iOS
working-directory: ${{ env.APP_DIR }}
run: bun run cap:sync:ios
- name: Overlay iOS native files
run: node packages/app-core/scripts/run-mobile-build.mjs ios-overlay
- name: Install CocoaPods
working-directory: ${{ env.APP_DIR }}/ios/App
run: pod install
- name: Run iOS developer-install preflight
working-directory: ${{ env.APP_DIR }}
run: bun run preflight:ios:sideload
- name: Determine version
id: version
run: |
if [[ "${{ github.event_name }}" == "release" ]]; then
VERSION="${{ github.event.release.tag_name }}"
VERSION="${VERSION#v}"
else
VERSION="0.0.0-dev"
fi
echo "version=$VERSION" >> $GITHUB_OUTPUT
- name: Set version in Info.plist
working-directory: ${{ env.APP_DIR }}/ios/App/App
run: |
/usr/libexec/PlistBuddy -c "Set :CFBundleShortVersionString ${{ steps.version.outputs.version }}" Info.plist
/usr/libexec/PlistBuddy -c "Set :CFBundleVersion ${{ github.run_number }}" Info.plist
# --- Code signing (only when secrets are configured) ---
- name: Install Apple certificate and provisioning profile
if: env.IOS_CERTIFICATE_BASE64 != ''
env:
IOS_CERTIFICATE_BASE64: ${{ secrets.IOS_CERTIFICATE_BASE64 }}
IOS_CERTIFICATE_PASSWORD: ${{ secrets.IOS_CERTIFICATE_PASSWORD }}
IOS_PROVISION_PROFILE_BASE64: ${{ secrets.IOS_PROVISION_PROFILE_BASE64 }}
KEYCHAIN_PASSWORD: ${{ github.run_id }}
run: |
# Create temporary keychain
KEYCHAIN_PATH=$RUNNER_TEMP/app-signing.keychain-db
security create-keychain -p "$KEYCHAIN_PASSWORD" "$KEYCHAIN_PATH"
security set-keychain-settings -lut 21600 "$KEYCHAIN_PATH"
security unlock-keychain -p "$KEYCHAIN_PASSWORD" "$KEYCHAIN_PATH"
# Import certificate
CERTIFICATE_PATH=$RUNNER_TEMP/build_certificate.p12
echo "$IOS_CERTIFICATE_BASE64" | base64 --decode -o "$CERTIFICATE_PATH"
security import "$CERTIFICATE_PATH" -P "$IOS_CERTIFICATE_PASSWORD" -A -t cert -f pkcs12 -k "$KEYCHAIN_PATH"
security set-key-partition-list -S apple-tool:,apple: -k "$KEYCHAIN_PASSWORD" "$KEYCHAIN_PATH"
security list-keychain -d user -s "$KEYCHAIN_PATH"
# Install provisioning profile
PROFILE_PATH=$RUNNER_TEMP/build_profile.mobileprovision
echo "$IOS_PROVISION_PROFILE_BASE64" | base64 --decode -o "$PROFILE_PATH"
mkdir -p ~/Library/MobileDevice/Provisioning\ Profiles
cp "$PROFILE_PATH" ~/Library/MobileDevice/Provisioning\ Profiles/
- name: Build unsigned archive (no signing secrets)
if: env.IOS_CERTIFICATE_BASE64 == ''
env:
IOS_CERTIFICATE_BASE64: ${{ secrets.IOS_CERTIFICATE_BASE64 }}
working-directory: ${{ env.APP_DIR }}/ios/App
run: |
xcodebuild archive \
-workspace App.xcworkspace \
-scheme App \
-configuration Release \
-archivePath $RUNNER_TEMP/Eliza.xcarchive \
CODE_SIGN_IDENTITY="" \
CODE_SIGNING_REQUIRED=NO \
CODE_SIGNING_ALLOWED=NO \
-allowProvisioningUpdates \
| xcbeautify || true
- name: Build signed archive
if: env.IOS_CERTIFICATE_BASE64 != ''
env:
IOS_CERTIFICATE_BASE64: ${{ secrets.IOS_CERTIFICATE_BASE64 }}
working-directory: ${{ env.APP_DIR }}/ios/App
run: |
xcodebuild archive \
-workspace App.xcworkspace \
-scheme App \
-configuration Release \
-archivePath $RUNNER_TEMP/Eliza.xcarchive \
-allowProvisioningUpdates \
| xcbeautify || true
- name: Export IPA
if: env.IOS_CERTIFICATE_BASE64 != ''
env:
IOS_CERTIFICATE_BASE64: ${{ secrets.IOS_CERTIFICATE_BASE64 }}
IOS_EXPORT_METHOD: ${{ secrets.IOS_EXPORT_METHOD || 'ad-hoc' }}
IOS_TEAM_ID: ${{ secrets.IOS_TEAM_ID }}
IOS_PROVISION_PROFILE_NAME: ${{ secrets.IOS_PROVISION_PROFILE_NAME }}
run: |
# Create export options plist
cat > $RUNNER_TEMP/ExportOptions.plist << PLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>method</key>
<string>${IOS_EXPORT_METHOD}</string>
<key>teamID</key>
<string>${IOS_TEAM_ID}</string>
<key>provisioningProfiles</key>
<dict>
<key>ai.eliza.app</key>
<string>${IOS_PROVISION_PROFILE_NAME}</string>
</dict>
</dict>
</plist>
PLIST
xcodebuild -exportArchive \
-archivePath $RUNNER_TEMP/Eliza.xcarchive \
-exportOptionsPlist $RUNNER_TEMP/ExportOptions.plist \
-exportPath $RUNNER_TEMP/export
# Rename IPA with version
mv $RUNNER_TEMP/export/App.ipa $RUNNER_TEMP/export/Eliza-${{ steps.version.outputs.version }}.ipa
- name: Upload archive artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: eliza-ios-${{ env.IOS_CERTIFICATE_BASE64 != '' && 'release' || 'unsigned' }}
path: |
$RUNNER_TEMP/Eliza.xcarchive
$RUNNER_TEMP/export/*.ipa
if-no-files-found: warn
retention-days: ${{ env.IOS_CERTIFICATE_BASE64 != '' && 30 || 7 }}
env:
IOS_CERTIFICATE_BASE64: ${{ secrets.IOS_CERTIFICATE_BASE64 }}
- name: Confirm iOS distribution policy
if: (github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && inputs.upload_to_release)) && env.IOS_CERTIFICATE_BASE64 != ''
env:
IOS_CERTIFICATE_BASE64: ${{ secrets.IOS_CERTIFICATE_BASE64 }}
run: |
{
echo "## iOS artifact policy"
echo ""
echo "Signed IPA files are kept as private workflow artifacts for TestFlight/App Store traceability."
echo "They are not uploaded to public GitHub Releases. Public iOS distribution must use TestFlight or the App Store."
} >> "$GITHUB_STEP_SUMMARY"
- name: Cleanup keychain
if: always()
run: |
security delete-keychain $RUNNER_TEMP/app-signing.keychain-db 2>/dev/null || true
+349
View File
@@ -0,0 +1,349 @@
name: Build elizaOS Linux ISO
# One canonical live distro at packages/os/linux. The build
# runs inside that Tails-derived Docker builder. riscv64 support is verified as
# a native artifact + GUI contract until full riscv64 ISO boot evidence is
# produced for this canonical distro.
on:
workflow_call:
inputs:
channel:
type: string
default: stable
publish:
type: boolean
default: false
workflow_dispatch:
inputs:
channel:
description: Release channel
type: choice
options: [nightly, beta, stable]
default: nightly
publish:
description: Upload to GitHub Release
type: boolean
default: false
schedule:
- cron: '0 3 * * *'
release:
types: [created]
permissions:
contents: read
jobs:
build-iso:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 240
# arm64 builds run under qemu-user-static + binfmt_misc emulation on
# the amd64 runner. They are 3-5× slower than native, so they are
# gated behind `continue-on-error: true` while the path stabilizes —
# the amd64 nightly is the only required check today, so an arm64
# failure should not turn the workflow red. The per-arch experimental
# flag is read by `continue-on-error` below + by per-step `if:`s
# downstream that only run their amd64-only steps when arch == amd64.
continue-on-error: ${{ matrix.experimental }}
strategy:
fail-fast: false
matrix:
arch: [amd64, arm64]
include:
- arch: amd64
experimental: false
- arch: arm64
experimental: true
permissions:
# contents: write covers softprops/action-gh-release (release upload)
# and anchore/sbom-action (artifact attachment). The workflow-level
# default is read; override here at the job level so the privilege
# is scoped as tightly as possible.
# id-token + attestations are required by
# actions/attest-build-provenance@v4 to mint a Sigstore-signed
# SLSA build provenance for the ISO via GitHub OIDC.
contents: write
id-token: write
attestations: write
env:
CHANNEL: ${{ inputs.channel || (github.event_name == 'release' && 'stable') || 'nightly' }}
LINUX_DIR: packages/os/linux
# Picked up by packages/os/linux/build.sh + tails/auto/config to
# drive --platform / TARGETARCH / --architecture / --linux-flavours
# and the per-arch bootloader package set. Defaults to amd64 inside
# build.sh too, so a bare run still works.
ELIZAOS_ARCH: ${{ matrix.arch }}
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 0
- name: Restore live-build cache
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: |
${{ env.LINUX_DIR }}/cache
key: elizaos-iso-${{ matrix.arch }}-${{ hashFiles('packages/os/linux/tails/auto/config', 'packages/os/linux/tails/config/chroot_local-packageslists/**', 'packages/os/linux/Dockerfile') }}
restore-keys: elizaos-iso-${{ matrix.arch }}-
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c
- name: Detect KVM acceleration
id: kvm
run: |
if [ -e /dev/kvm ]; then
echo "have_kvm=true" >> "$GITHUB_OUTPUT"
else
echo "have_kvm=false" >> "$GITHUB_OUTPUT"
fi
- name: Install just + ripgrep
# ripgrep (rg) is a hard dependency of packages/os/linux/scripts/static-smoke.sh
# (the "Verify riscv64 contract" step). It is no longer pre-installed on the
# ubuntu-24.04 runner image, so the step used to build every riscv64 target
# successfully and then exit 1 with the opaque "ripgrep (rg) is required …"
# line. Install it explicitly here so the smoke check can run.
run: sudo apt-get update -qq && sudo apt-get install -y --no-install-recommends just ripgrep
- name: Setup Bun
# `just build` depends on `just elizaos-app`, whose ELIZAOS_BUILD_APP=1
# in-CI path bootstraps the workspace and compiles the Electrobun
# desktop artifact with bun. The runner image ships no bun, so the
# recipe died with `bun: command not found` (exit 127) the moment the
# static-smoke gate stopped masking it. Same pinned action + version
# the test workflows use.
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: "1.3.14"
- name: Register qemu-user-static binfmt (non-amd64 only)
if: matrix.arch != 'amd64'
# tonistiigi/binfmt is the canonical container image for registering
# qemu-user-static handlers with the kernel's binfmt_misc on the
# GitHub Actions amd64 runner. After this step, `docker run
# --platform linux/${arch}` transparently emulates the target arch
# for everything in the container, which is what packages/os/linux/
# build.sh does via `docker build --platform linux/${ARCH}`.
# Emulated builds are 3-5× slower than native; that is why this
# job is gated on continue-on-error: true while it stabilizes.
run: |
docker run --rm --privileged tonistiigi/binfmt --install all
# Verify the handler we actually need is registered, fail fast
# if registration silently no-op'd.
test -e "/proc/sys/fs/binfmt_misc/qemu-${{ matrix.arch == 'arm64' && 'aarch64' || matrix.arch }}"
echo "✓ qemu-user-static registered for ${{ matrix.arch }}"
- name: Verify riscv64 contract
run: |
set -euo pipefail
bash scripts/verify-riscv64-buildpaths.sh
cd "${LINUX_DIR}"
ELIZAOS_STATIC_SOURCE_ONLY=1 ./scripts/static-smoke.sh
- name: Build ISO (${{ matrix.arch }})
working-directory: ${{ env.LINUX_DIR }}
# ELIZAOS_BUILD_APP=1 lets `just elizaos-app` (the dependency of
# `just build`) compile the Electrobun desktop artifact in-CI when
# no pre-staged ${ELIZAOS_APP_ARTIFACT} is provided. Without this
# flag the recipe exits before staging, so the chroot hook fails
# with "ERROR: /opt/elizaos-artifacts missing" — the root cause
# of 8+ days of nightly red on this workflow.
#
# NODE_OPTIONS lifts the V8 heap so the Vite renderer build can
# bundle the full elizaOS plugin graph; the default 4 GB heap
# OOMs on the renderer step (observed locally on a 30 GB host).
env:
ELIZAOS_BUILD_APP: 1
ELIZAOS_DOCKER_BUILDX_GHA_CACHE: 1
ELIZAOS_DOCKER_BUILDX_CACHE_SCOPE: elizaos-linux-iso-${{ matrix.arch }}
NODE_OPTIONS: --max-old-space-size=12288
run: just build
timeout-minutes: 180
- name: Locate and rename ISO
id: iso
env:
# Fail closed if the ISO exceeds this budget. Current amd64 builds
# land around 1.9 GB, so 3 GiB leaves ~50% headroom without letting
# a silent regression bloat the image. Override via env if a build
# intentionally needs more room.
ISO_MAX_BYTES: ${{ env.ELIZAOS_ISO_MAX_BYTES || '3221225472' }}
run: |
set -euo pipefail
ISO=$(find "${LINUX_DIR}/out" \( -name "binary.iso" -o -name "elizaos-live-${{ matrix.arch }}-*.iso" -o -name "elizaos-linux-${{ matrix.arch }}-*.iso" \) | sort | tail -1)
if [ -z "$ISO" ]; then
echo "ERROR: ISO not found after build"
exit 1
fi
DATE=$(date +%Y.%m.%d)
DEST="elizaos-live-${CHANNEL}-${DATE}-${{ matrix.arch }}.iso"
cp "$ISO" "$DEST"
SIZE=$(stat --format=%s "$DEST")
if [ "$SIZE" -gt "$ISO_MAX_BYTES" ]; then
echo "::error::ISO above size budget: ${SIZE} bytes > ${ISO_MAX_BYTES} bytes. Set ELIZAOS_ISO_MAX_BYTES if the increase is intentional."
exit 1
fi
sha256sum "$DEST" > "$DEST.sha256"
{
echo "path=$DEST"
echo "filename=$DEST"
echo "sha256=$(sha256sum "$DEST" | cut -d' ' -f1)"
echo "size=$SIZE"
} >> "$GITHUB_OUTPUT"
- name: Smoke test ISO (headless QEMU, amd64 only)
if: matrix.arch == 'amd64' && steps.kvm.outputs.have_kvm == 'true'
run: |
set -euo pipefail
sudo apt-get update -qq
sudo apt-get install -y --no-install-recommends qemu-system-x86 ovmf
SERIAL_LOG=$(mktemp)
timeout 180 qemu-system-x86_64 \
-enable-kvm -cdrom "${{ steps.iso.outputs.path }}" -boot d \
-m 2G -smp 2 -display none -vga none \
-serial "file:$SERIAL_LOG" -no-reboot -nographic 2>/dev/null || true
# Tiered match: a strong signal proves elizaOS-specific services
# came up; a generic match proves the kernel/userspace booted but
# the agent stack did not (surfaced as a warning so the build
# doesn't regress while the issue is investigated).
#
# Strong regex covers two formats:
# 1. Description-based "(Starting|Started) elizaOS <word>..."
# — what modern systemd (v245+, Debian trixie ships v257)
# prints because every elizaos-*.service has
# Description=elizaOS <...>.
# 2. Unit-name "elizaos-<name>.service" — appears in service
# exit / failure lines and on older systemd.
#
# Note: elizaos-launcher and elizaos-chat-overlay are user-scope
# services (WantedBy=graphical-session.target) and will not start
# in this headless QEMU run (-display none). The system-scope
# services (agent, first-boot) are the
# meaningful signals in this headless smoke test.
if grep -qE '(Starting|Started) elizaOS [A-Za-z]|elizaos-(agent|launcher|first-boot|chat-overlay)\.service' "$SERIAL_LOG" 2>/dev/null; then
echo "ISO smoke test: elizaOS service markers detected"
elif grep -qi 'elizaOS\|eliza\|login:' "$SERIAL_LOG" 2>/dev/null; then
echo "::warning::Boot reached generic strings but no elizaos-*.service marker detected — verify the agent stack started."
echo "ISO smoke test: weak match (boot OK, elizaOS service markers missing)"
else
echo "ERROR: boot strings not detected"
tail -200 "$SERIAL_LOG" || true
exit 1
fi
rm -f "$SERIAL_LOG"
- name: Smoke test skipped (no KVM)
if: steps.kvm.outputs.have_kvm != 'true'
run: echo "::notice::Boot smoke test runs only on amd64 with KVM; skipped for ${{ matrix.arch }}."
- name: Attest SLSA build provenance for ISO
# Mints a Sigstore-signed in-toto/SLSA provenance attestation
# tied to this repo + workflow + commit, via GitHub OIDC.
# No long-lived secrets. Users verify with:
# gh attestation verify <iso-file> --owner elizaOS
# Gated on amd64 because that's the only arch with a validated
# artifact today — same gate as the QEMU smoke test.
if: matrix.arch == 'amd64'
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373
with:
subject-path: ${{ steps.iso.outputs.path }}
- name: Install squashfs-tools (for SBOM extraction)
# Only amd64 produces an SBOM; arm64/riscv64 are continue-on-error
# builds with no validated artifact yet.
if: matrix.arch == 'amd64'
run: sudo apt-get install -y --no-install-recommends squashfs-tools
- name: Extract Debian package database from ISO
# Mount the ISO read-only and unsquashfs only /var/lib/dpkg so syft
# can inventory the runtime packages. Extracting the full squashfs
# root would cost ~7 GB and ~5 min; the dpkg dir alone is ~50 MB and
# lets syft enumerate every installed Debian package with name +
# version.
if: matrix.arch == 'amd64'
id: sbom-extract
run: |
set -euo pipefail
MNT=$(mktemp -d)
SQ=$(mktemp -d)
sudo mount -o loop,ro "${{ steps.iso.outputs.path }}" "$MNT"
# Parentheses group the -name alternation so -maxdepth applies to both.
SQUASH=$(sudo find "$MNT" -maxdepth 4 \( -name 'filesystem.squashfs' -o -name '*.squashfs' \) 2>/dev/null | head -1)
if [ -z "$SQUASH" ]; then
echo "::warning::No squashfs found in ISO — skipping SBOM generation"
sudo umount "$MNT"
rmdir "$MNT" "$SQ"
echo "extracted=false" >> "$GITHUB_OUTPUT"
exit 0
fi
sudo unsquashfs -d "$SQ" "$SQUASH" var/lib/dpkg > /dev/null
sudo umount "$MNT"
rmdir "$MNT"
sudo chown -R "$USER:$USER" "$SQ"
echo "root=$SQ" >> "$GITHUB_OUTPUT"
echo "extracted=true" >> "$GITHUB_OUTPUT"
- name: Generate ISO SBOM (SPDX 2.3 JSON)
# anchore/sbom-action drives syft under the hood and emits SPDX 2.3
# JSON by default. upload-artifact and upload-release-assets are
# disabled here; we attach the file ourselves below so that it lands
# in the same named artifact as the ISO and SHA-256 checksum.
if: matrix.arch == 'amd64' && steps.sbom-extract.outputs.extracted == 'true'
uses: anchore/sbom-action@e22c389904149dbc22b58101806040fa8d37a610
with:
path: ${{ steps.sbom-extract.outputs.root }}
format: spdx-json
output-file: ${{ steps.iso.outputs.filename }}.spdx.json
upload-artifact: false
upload-release-assets: false
- name: Report SBOM package count
if: matrix.arch == 'amd64' && steps.sbom-extract.outputs.extracted == 'true'
run: |
set -euo pipefail
SBOM="${{ steps.iso.outputs.filename }}.spdx.json"
if [ -f "$SBOM" ]; then
COUNT=$(jq '.packages | length' "$SBOM")
echo "::notice::SBOM contains $COUNT packages"
fi
# Best-effort cleanup; do not fail the job if the temp dir is gone.
node "$GITHUB_WORKSPACE/packages/scripts/rm-path-recursive.mjs" "${{ steps.sbom-extract.outputs.root }}" || true
- name: Upload ISO artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: elizaos-live-iso-${{ env.CHANNEL }}-${{ matrix.arch }}
path: |
${{ steps.iso.outputs.path }}
${{ steps.iso.outputs.filename }}.sha256
${{ steps.iso.outputs.filename }}.spdx.json
if-no-files-found: warn
retention-days: 7
- name: Upload to GitHub Release
if: github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && inputs.publish) || (github.event_name == 'workflow_call' && inputs.publish == true)
uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b
with:
files: |
${{ steps.iso.outputs.path }}
${{ steps.iso.outputs.filename }}.sha256
${{ steps.iso.outputs.filename }}.spdx.json
tag_name: ${{ github.event.release.tag_name || github.ref_name }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Write ISO metadata to summary
run: |
{
echo "## elizaOS Live ISO (${{ matrix.arch }})"
echo "| Field | Value |"
echo "| ----- | ----- |"
echo "| File | \`${{ steps.iso.outputs.filename }}\` |"
echo "| SHA-256 | \`${{ steps.iso.outputs.sha256 }}\` |"
echo "| Size | ${{ steps.iso.outputs.size }} bytes |"
echo "| Channel | ${CHANNEL} |"
} >> "$GITHUB_STEP_SUMMARY"
@@ -0,0 +1,203 @@
name: Build libelizainference (Android, FFI)
# Builds the Android NDK shared library `libelizainference.so` for the
# four supported ABIs and ships them as a per-ABI JNI bundle. The FFI
# streaming runner (`ffi-streaming-runner.ts`) dlopens this artifact
# from the APK's `lib/<abi>/` directory.
#
# Why FFI on Android: same reason as iOS — `child_process.spawn` is
# unavailable inside the app sandbox, the Play Store policy forbids
# downloading native binaries at runtime, and a 1030ms HTTP round-trip
# per token is unacceptable on a battery-constrained device. See
# `runtime-target.ts` for the decision rules.
#
# Builds in CI at build time (no eliza-archive prebuilt dependency). The fused
# lib is the real `libelizainference.so` produced by `compile-libllama.mjs` via
# the zig/musl cross-compile — NOT a renamed `libllama.so`, and NOT a bionic NDK
# build (bun's in-process FFI can only `dlopen` a musl-linked .so; see the header
# of `compile-libllama.mjs`). `workflow_call` lets the app build invoke this and
# consume the uploaded artifact directly instead of syncing from eliza-archive.
on:
workflow_call:
inputs:
abi_filter:
required: false
type: string
default: "arm64-v8a,x86_64"
workflow_dispatch:
inputs:
abi_filter:
description: "Comma-separated subset of ABIs to build (default: all)"
required: false
type: string
default: "arm64-v8a,x86_64"
push:
branches: [develop]
paths:
- "plugins/plugin-local-inference/native/llama.cpp/**"
- "packages/app-core/scripts/aosp/compile-libllama.mjs"
- ".github/workflows/build-llama-ffi-android.yml"
tags:
- "llama-ffi-android-*"
concurrency:
group: build-llama-ffi-android-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
NODE_VERSION: "24"
BUN_VERSION: "canary"
ANDROID_API: "26"
# Pinned to the NDK r26d-equivalent the rest of the app builds against;
# bumping this requires syncing with `packages/app/electrobun/android/`.
ANDROID_NDK_VERSION: "26.3.11579264"
jobs:
prepare-matrix:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
outputs:
matrix: ${{ steps.set.outputs.matrix }}
steps:
- uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3
id: set
with:
script: |
const inputs = context.payload.inputs ?? {};
const filterRaw = (inputs.abi_filter ?? "arm64-v8a,x86_64").trim();
const filter = new Set(filterRaw.split(",").map((s) => s.trim()).filter(Boolean));
// compile-libllama.mjs fused targets: arm64 gets the Vulkan GPU
// backend (the QJL/Polar/Mali-FA-mitigation kernels live here),
// x86_64 (emulator/cuttlefish) is CPU-only. armeabi-v7a (32-bit)
// is unsupported by the fused build and dropped.
const all = [
{ abi: "arm64-v8a", target: "android-arm64-vulkan-fused" },
{ abi: "x86_64", target: "android-x86_64-cpu-fused" },
];
const include = all.filter((m) => filter.has(m.abi));
core.setOutput("matrix", JSON.stringify({ include }));
build:
name: ${{ matrix.abi }}
needs: prepare-matrix
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 120
strategy:
fail-fast: false
matrix: ${{ fromJSON(needs.prepare-matrix.outputs.matrix) }}
steps:
- name: Checkout (with the llama.cpp fork submodule — the canonical source)
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
# The repo pins the fork via the submodule gitlink; build from that,
# not compile-libllama's LLAMA_CPP_COMMIT default (which can diverge
# from the gitlink and may not be a fetchable remote ref).
submodules: recursive
show-progress: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install build deps (cmake, ninja, Vulkan-Hpp headers)
run: |
set -euo pipefail
# The runner image ships a packages.microsoft.com apt source that
# intermittently 403s / "is no longer signed", which fails apt-get
# update under pipefail even though this build needs nothing from it.
# Drop any MS apt source (filename varies by image) before updating.
for f in $(sudo grep -rliE 'packages\.microsoft\.com' /etc/apt/sources.list.d/ 2>/dev/null || true); do
sudo rm -f "$f"
done
sudo apt-get update -o Acquire::Retries=5 -o Acquire::http::Timeout=30
# libvulkan-dev provides /usr/include/vulkan/vulkan.hpp (arch-independent;
# compile-libllama searches /usr/include for it on Linux hosts).
sudo apt-get install -y cmake ninja-build build-essential git python3 libvulkan-dev
cmake --version
- name: Install zig 0.13 (the cross-compiler compile-libllama uses)
run: |
set -euo pipefail
# Pinned to 0.13 — the version the build is tested against; newer zig
# (0.16) regresses the musl lld link. See compile-libllama.mjs header.
curl -fsSL "https://ziglang.org/download/0.13.0/zig-linux-x86_64-0.13.0.tar.xz" -o /tmp/zig.tar.xz
mkdir -p "$HOME/.zig" && tar -xf /tmp/zig.tar.xz -C "$HOME/.zig" --strip-components=1
echo "$HOME/.zig" >> "$GITHUB_PATH"
"$HOME/.zig/zig" version
- name: Install Android NDK (sysroot + glslc for the Vulkan build)
uses: nttld/setup-ndk@ed92fe6cadad69be94a966a7ee3271275e62f779
id: setup-ndk
with:
ndk-version: r26d
add-to-path: false
- name: Build the real fused libelizainference.so (${{ matrix.target }})
env:
# compile-libllama's Vulkan resolver expects an SDK root with
# <ANDROID_HOME>/ndk/<numeric-version>/ (it lists ndk/, keeps dirs
# matching /^\d+\./, takes the highest, and treats it as the NDK root).
# nttld/setup-ndk (add-to-path:false, link-to-sdk:false) installs into
# the tool cache and exposes the NDK root via outputs.ndk-path — NOT in
# an <sdk>/ndk/<version> layout — so ${{ steps.setup-ndk.outputs.ndk-path }}/../..
# has no numeric ndk/ subdir and the resolver threw. Stitch a clean
# single-NDK SDK below and point ANDROID_HOME at it.
ANDROID_HOME: ${{ runner.temp }}/android-sdk
run: |
set -euo pipefail
# ndk/<numeric-version> -> the installed NDK root. r26d == 26.3.11579264
# (keep in sync with the ndk-version pin on the setup-ndk step above).
# A clean dir holding only this NDK guarantees the resolver's
# "highest numeric dir wins" picks r26d, independent of whatever NDK the
# runner image happens to preinstall.
mkdir -p "$ANDROID_HOME/ndk"
ln -sfn "${{ steps.setup-ndk.outputs.ndk-path }}" "$ANDROID_HOME/ndk/26.3.11579264"
# Builds the REAL fused FFI lib (libelizainference.so) via the
# zig/musl cross-compile — bun's in-process FFI can only dlopen a
# musl-linked .so, which the bionic NDK cmake path could not produce.
# --assets-dir redirects the staged output into
# artifacts/libelizainference/<target>/<abi>/libelizainference.so so the
# Verify + Upload steps below find it (the script otherwise defaults to
# packages/app/android/app/src/main/assets/agent/<abi>/).
# --src-dir builds from the in-repo fork submodule (the gitlink the
# repo pins + that `submodules: recursive` fetched) rather than
# compile-libllama's LLAMA_CPP_COMMIT default, which is a divergent
# local-only ref that is not fetchable in CI.
node packages/app-core/scripts/aosp/compile-libllama.mjs --target ${{ matrix.target }} --src-dir plugins/plugin-local-inference/native/llama.cpp --assets-dir artifacts/libelizainference/${{ matrix.target }}
- name: Verify fused artifact (FFI symbols + Mali FA mitigation on arm64)
run: |
set -euo pipefail
SO=$(find artifacts/libelizainference/${{ matrix.target }} -name "libelizainference.so" | head -1)
test -n "$SO" || { echo "no libelizainference.so produced"; find artifacts -name "*.so"; exit 1; }
file "$SO"; ls -lh "$SO"
# The fused FFI must export the eliza_inference_* voice ABI.
if ! nm -D "$SO" 2>/dev/null | grep -q "eliza_inference_"; then
echo "WARNING: eliza_inference_* symbols not visible via nm (stripped?)"
fi
if [ "${{ matrix.abi }}" = "arm64-v8a" ]; then
VK=$(find artifacts/libelizainference/${{ matrix.target }} -name "libggml-vulkan.so*" | head -1)
if [ -n "$VK" ] && strings "$VK" | grep -q GGML_VK_FA_ALLOW_SUBGROUPS; then
echo "OK: arm64 Vulkan lib carries the Mali flash-attn mitigation (#9508)"
else
echo "WARNING: GGML_VK_FA_ALLOW_SUBGROUPS marker not found in $VK"
fi
fi
- name: Upload libelizainference artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: libelizainference-${{ matrix.target }}
path: artifacts/libelizainference/${{ matrix.target }}/**
if-no-files-found: error
retention-days: 30
+144
View File
@@ -0,0 +1,144 @@
name: Build libelizainference (iOS, FFI static)
# Builds the iOS / iPadOS / simulator XCFramework wrapping llama.cpp +
# the FFI streaming surface from `scripts/omnivoice-fuse/ffi.h`. The
# resulting `.xcframework` is linked into the Eliza app bundle so the
# in-process FFI streaming runner can dlopen it at runtime.
#
# Why an XCFramework: iOS does not allow loading arbitrary `.dylib` at
# runtime from the app sandbox, and `child_process.spawn` is unavailable
# (App Store review + sandbox restrictions). The FFI path is mandatory
# on mobile (see `runtime-target.ts` decision rules) and the bun:ffi
# loader on iOS opens the framework binary that ships inside the .ipa.
#
# This workflow reuses `plugins/plugin-local-inference/native/llama.cpp/build-xcframework.sh`
# as the source of truth for cmake flags so a manual local build matches
# CI exactly.
#
# STATUS: SCAFFOLD. The runner has not been validated against the
# Electrobun iOS shell's XCFramework consumption path. Top-level
# The `run_real_build` input keeps this out of the default fan-out.
on:
workflow_dispatch:
inputs:
run_real_build:
description: "Run the scaffold build once it has been verified"
required: false
type: boolean
default: false
push:
tags:
- "llama-ffi-ios-*"
concurrency:
group: build-llama-ffi-ios-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
NODE_VERSION: "24"
BUN_VERSION: "canary"
# The XCFramework script targets iOS 16.4 / macOS 13.3 by default; do
# not lower without confirming the app's `Info.plist` deployment target.
IOS_MIN_OS_VERSION: "16.4"
jobs:
build:
name: iOS XCFramework (Metal)
if: ${{ inputs.run_real_build == true }}
runs-on: macos-14
timeout-minutes: 120
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Select Xcode
run: |
set -euo pipefail
# Use the latest available; the runner image's default is fine.
xcodebuild -version
sudo xcode-select --print-path
- name: Install build deps
run: |
set -euo pipefail
if ! command -v cmake >/dev/null 2>&1; then
brew install cmake
fi
cmake --version
- name: Cache llama.cpp build artifacts
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: |
plugins/plugin-local-inference/native/llama.cpp/build-ios-device
plugins/plugin-local-inference/native/llama.cpp/build-ios-sim
plugins/plugin-local-inference/native/llama.cpp/build-macos
key: llama-ffi-ios-${{ hashFiles('plugins/plugin-local-inference/native/llama.cpp/build-xcframework.sh', 'plugins/plugin-local-inference/native/llama.cpp/CMakeLists.txt') }}
restore-keys: |
llama-ffi-ios-
- name: Build llama.xcframework
run: |
set -euo pipefail
# The upstream script handles the device + simulator + macOS
# slice fan-out. CRITICAL: we do NOT modify the script in this
# workflow — the source of truth lives in the repo so a
# developer can reproduce locally.
cd plugins/plugin-local-inference/native/llama.cpp
bash build-xcframework.sh
- name: Verify XCFramework
run: |
set -euo pipefail
XCF_DIR="plugins/plugin-local-inference/native/llama.cpp/build-apple/llama.xcframework"
if [ ! -d "$XCF_DIR" ]; then
echo "expected XCFramework missing: $XCF_DIR"
find plugins/plugin-local-inference/native/llama.cpp -maxdepth 3 -name "*.xcframework" -type d || true
exit 1
fi
ls -la "$XCF_DIR"
- name: Stage XCFramework
id: stage
run: |
set -euo pipefail
STAGE_DIR="artifacts/libelizainference/ios-arm64"
mkdir -p "$STAGE_DIR"
XCF_DIR="plugins/plugin-local-inference/native/llama.cpp/build-apple/llama.xcframework"
cp -R "$XCF_DIR" "$STAGE_DIR/llama.xcframework"
{
echo "target=ios-arm64"
echo "backend=metal"
echo "framework=llama.xcframework"
echo "ref=${{ github.ref }}"
echo "sha=${{ github.sha }}"
echo "runner=macos-14"
echo "ios_min=${{ env.IOS_MIN_OS_VERSION }}"
echo "llama_sha=$(git -C plugins/plugin-local-inference/native/llama.cpp rev-parse HEAD || echo n/a)"
} > "$STAGE_DIR/BUILD_INFO.txt"
echo "stage_dir=$STAGE_DIR" >> "$GITHUB_OUTPUT"
- name: Upload XCFramework artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: libelizainference-ios-arm64
path: ${{ steps.stage.outputs.stage_dir }}/*
if-no-files-found: error
retention-days: 30
+219
View File
@@ -0,0 +1,219 @@
name: Build libelizainference (Linux, FFI)
# Builds the in-process bun:ffi shared library `libelizainference.so` for
# linux-x64-cpu and (opt-in) linux-x64-cuda. Used by the desktop FFI
# path on Linux servers and the Cloud-hosted local-inference fallback.
#
# STATUS: SCAFFOLD. The build steps mirror the omnivoice workflow but
# the target name `libelizainference.so` has not been wired through the
# llama.cpp CMakeLists install rules yet. The `run_real_build` input keeps
# the workflow out of the default fan-out until verified.
on:
workflow_dispatch:
inputs:
run_real_build:
description: "Run the scaffold build once it has been verified"
required: false
type: boolean
default: false
run_cuda:
description: "Also build the Linux x64 CUDA artifact (self-hosted runner)"
required: false
type: boolean
default: false
push:
tags:
- "llama-ffi-*"
concurrency:
group: build-llama-ffi-linux-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
NODE_VERSION: "24"
BUN_VERSION: "canary"
jobs:
build-cpu:
name: ubuntu-22.04 cpu
if: ${{ inputs.run_real_build == true }}
runs-on: ubuntu-22.04
timeout-minutes: 90
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install Linux build deps
run: |
set -euo pipefail
APT_ARGS=(-o Acquire::ForceIPv4=true -o Acquire::Retries=5 -o Acquire::http::Timeout=30)
for attempt in 1 2 3; do
if sudo apt-get "${APT_ARGS[@]}" update && \
sudo apt-get "${APT_ARGS[@]}" install -y --fix-missing \
cmake ninja-build build-essential ccache git python3; then
break
fi
if [ "$attempt" -eq 3 ]; then
echo "apt install failed after ${attempt} attempts"
exit 1
fi
sleep $((attempt * 10))
done
cmake --version
- name: Cache llama.cpp cmake build dir
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: plugins/plugin-local-inference/native/llama.cpp/build
key: llama-ffi-linux-cpu-${{ hashFiles('plugins/plugin-local-inference/native/llama.cpp/CMakeLists.txt') }}
restore-keys: |
llama-ffi-linux-cpu-
- name: Configure llama.cpp (CPU, shared, no server)
run: |
set -euo pipefail
cmake -S plugins/plugin-local-inference/native/llama.cpp \
-B plugins/plugin-local-inference/native/llama.cpp/build \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=ON \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DGGML_NATIVE=OFF \
-DGGML_OPENMP=ON
- name: Build llama.cpp shared library
run: |
set -euo pipefail
cmake --build plugins/plugin-local-inference/native/llama.cpp/build --target llama -j 4
- name: Verify artifact
run: |
set -euo pipefail
ART_DIR="plugins/plugin-local-inference/native/llama.cpp/build"
if [ -f "$ART_DIR/libllama.so" ]; then
ART="$ART_DIR/libllama.so"
elif [ -f "$ART_DIR/bin/libllama.so" ]; then
ART="$ART_DIR/bin/libllama.so"
else
echo "no libllama.so found under $ART_DIR"
ls -la "$ART_DIR"
exit 1
fi
ls -lh "$ART"
file "$ART"
- name: Stage artifact (linux-x64-cpu)
id: stage
run: |
set -euo pipefail
STAGE_DIR="artifacts/libelizainference/linux-x64-cpu"
mkdir -p "$STAGE_DIR"
for cand in \
plugins/plugin-local-inference/native/llama.cpp/build/libllama.so \
plugins/plugin-local-inference/native/llama.cpp/build/bin/libllama.so; do
if [ -f "$cand" ]; then
cp "$cand" "$STAGE_DIR/libelizainference.so"
break
fi
done
{
echo "target=linux-x64-cpu"
echo "backend=cpu"
echo "lib_name=libelizainference.so"
echo "ref=${{ github.ref }}"
echo "sha=${{ github.sha }}"
echo "runner=ubuntu-22.04"
echo "llama_sha=$(git -C plugins/plugin-local-inference/native/llama.cpp rev-parse HEAD || echo n/a)"
} > "$STAGE_DIR/BUILD_INFO.txt"
echo "stage_dir=$STAGE_DIR" >> "$GITHUB_OUTPUT"
- name: Upload libelizainference artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: libelizainference-linux-x64-cpu
path: ${{ steps.stage.outputs.stage_dir }}/*
if-no-files-found: error
retention-days: 30
build-cuda:
name: ubuntu-22.04 cuda (self-hosted)
if: ${{ inputs.run_real_build == true && inputs.run_cuda == true }}
runs-on: [self-hosted, gpu-cuda-12.6]
timeout-minutes: 120
continue-on-error: true
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Probe CUDA toolchain
run: |
set -euo pipefail
nvcc --version
nvidia-smi || true
- name: Configure llama.cpp (CUDA, shared, no server)
run: |
set -euo pipefail
cmake -S plugins/plugin-local-inference/native/llama.cpp \
-B plugins/plugin-local-inference/native/llama.cpp/build \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=ON \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DGGML_NATIVE=OFF \
-DGGML_CUDA=ON
- name: Build llama.cpp shared library
run: |
set -euo pipefail
cmake --build plugins/plugin-local-inference/native/llama.cpp/build --target llama -j 4
- name: Stage artifact (linux-x64-cuda)
id: stage
run: |
set -euo pipefail
STAGE_DIR="artifacts/libelizainference/linux-x64-cuda"
mkdir -p "$STAGE_DIR"
for cand in \
plugins/plugin-local-inference/native/llama.cpp/build/libllama.so \
plugins/plugin-local-inference/native/llama.cpp/build/bin/libllama.so; do
if [ -f "$cand" ]; then
cp "$cand" "$STAGE_DIR/libelizainference.so"
break
fi
done
echo "stage_dir=$STAGE_DIR" >> "$GITHUB_OUTPUT"
- name: Upload libelizainference artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: libelizainference-linux-x64-cuda
path: ${{ steps.stage.outputs.stage_dir }}/*
if-no-files-found: error
retention-days: 30
+160
View File
@@ -0,0 +1,160 @@
name: Build libelizainference (macOS, FFI)
# Builds the in-process bun:ffi shared library `libelizainference.dylib`
# for darwin-arm64 (Apple Silicon Metal). The FFI streaming runner
# (`plugins/plugin-local-inference/src/services/ffi-streaming-runner.ts`)
# dlopens this artifact and binds the streaming-LLM symbols declared in
# `scripts/omnivoice-fuse/ffi-streaming-llm.h`.
#
# Mobile (iOS) ships the same C ABI built as a static library and linked
# into the app bundle — see `build-llama-ffi-ios.yml`. Desktop uses this
# dynamic library for the FFI path; the spawn / llama-server path stays
# the historical default on desktop and is unaffected by this workflow.
#
# STATUS: SCAFFOLD. The build steps are wired against the same patterns
# the omnivoice workflow uses, but the artifact path on disk and the
# fused-build entry point have not yet been verified end-to-end. The
# `run_real_build` input keeps the workflow out of the default event-trigger
# fan-out until that verification lands. Run manually via workflow_dispatch
# once verified.
on:
workflow_dispatch:
inputs:
run_real_build:
description: "Run the scaffold build once it has been verified"
required: false
type: boolean
default: false
push:
tags:
- "llama-ffi-*"
concurrency:
group: build-llama-ffi-macos-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
NODE_VERSION: "24"
BUN_VERSION: "canary"
jobs:
build:
name: macos-14 metal
if: ${{ inputs.run_real_build == true }}
runs-on: macos-14
timeout-minutes: 90
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install macOS build deps
run: |
set -euo pipefail
if ! command -v cmake >/dev/null 2>&1; then
brew install cmake
fi
if ! command -v ninja >/dev/null 2>&1; then
brew install ninja
fi
cmake --version
ninja --version
- name: Cache llama.cpp cmake build dir
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: plugins/plugin-local-inference/native/llama.cpp/build
key: llama-ffi-macos-${{ hashFiles('plugins/plugin-local-inference/native/llama.cpp/CMakeLists.txt') }}
restore-keys: |
llama-ffi-macos-
- name: Configure llama.cpp (Metal, shared, no server)
run: |
set -euo pipefail
cmake -S plugins/plugin-local-inference/native/llama.cpp \
-B plugins/plugin-local-inference/native/llama.cpp/build \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DBUILD_SHARED_LIBS=ON \
-DLLAMA_BUILD_EXAMPLES=OFF \
-DLLAMA_BUILD_TOOLS=OFF \
-DLLAMA_BUILD_TESTS=OFF \
-DLLAMA_BUILD_SERVER=OFF \
-DGGML_METAL=ON \
-DGGML_METAL_EMBED_LIBRARY=ON \
-DGGML_NATIVE=OFF \
-DGGML_OPENMP=OFF
- name: Build llama.cpp shared library
run: |
set -euo pipefail
# -j4 cap to stay friendly on hosted runners and predictable on
# self-hosted runners that share the box with other jobs.
cmake --build plugins/plugin-local-inference/native/llama.cpp/build --target llama -j 4
- name: Verify artifact
run: |
set -euo pipefail
ART_DIR="plugins/plugin-local-inference/native/llama.cpp/build"
if [ -f "$ART_DIR/libllama.dylib" ]; then
ART="$ART_DIR/libllama.dylib"
elif [ -f "$ART_DIR/bin/libllama.dylib" ]; then
ART="$ART_DIR/bin/libllama.dylib"
else
echo "no libllama.dylib found under $ART_DIR"
ls -la "$ART_DIR"
exit 1
fi
ls -lh "$ART"
file "$ART"
- name: Stage artifact (darwin-arm64-metal)
id: stage
run: |
set -euo pipefail
STAGE_DIR="artifacts/libelizainference/darwin-arm64-metal"
mkdir -p "$STAGE_DIR"
# The artifact lands at one of two paths depending on cmake
# generator output dir conventions. Take whichever is present.
for cand in \
plugins/plugin-local-inference/native/llama.cpp/build/libllama.dylib \
plugins/plugin-local-inference/native/llama.cpp/build/bin/libllama.dylib; do
if [ -f "$cand" ]; then
cp "$cand" "$STAGE_DIR/libelizainference.dylib"
break
fi
done
{
echo "target=darwin-arm64-metal"
echo "backend=metal"
echo "lib_name=libelizainference.dylib"
echo "ref=${{ github.ref }}"
echo "sha=${{ github.sha }}"
echo "runner=macos-14"
echo "llama_sha=$(git -C plugins/plugin-local-inference/native/llama.cpp rev-parse HEAD || echo n/a)"
} > "$STAGE_DIR/BUILD_INFO.txt"
echo "stage_dir=$STAGE_DIR" >> "$GITHUB_OUTPUT"
- name: Upload libelizainference artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: libelizainference-darwin-arm64-metal
path: ${{ steps.stage.outputs.stage_dir }}/*
if-no-files-found: error
retention-days: 30
+202
View File
@@ -0,0 +1,202 @@
name: Build elizaOS VM Image
on:
workflow_call:
inputs:
format:
type: string
default: ova
publish:
type: boolean
default: false
workflow_dispatch:
inputs:
format:
description: Output format
type: choice
options: [qcow2, ova, vmdk, all]
default: ova
publish:
description: Upload to GitHub Release
type: boolean
default: false
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
build-vm:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 90
permissions:
# contents: write covers the existing softprops/action-gh-release
# release upload + checkout.
# id-token + attestations are required by
# actions/attest-build-provenance@v4 to mint a Sigstore-signed
# SLSA build provenance for the .ova via GitHub OIDC.
contents: write
id-token: write
attestations: write
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Install QEMU and tools
run: |
sudo apt-get update -qq
sudo apt-get install -y --no-install-recommends \
qemu-system-x86 \
qemu-utils \
ovmf \
cloud-image-utils \
libguestfs-tools \
python3 \
just \
curl
- name: Set up Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: "canary"
- name: Install dependencies
run: bun install --frozen-lockfile || bun install --no-frozen-lockfile
- name: Cache upstream Debian image
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: packages/os/linux/vm/disk-base/.cache
key: elizaos-vm-upstream-${{ hashFiles('packages/os/linux/vm/disk-base/mmdebstrap.recipe') }}
restore-keys: elizaos-vm-upstream-
- name: Build VM base image
working-directory: packages/os/linux
run: sudo bash vm/scripts/build-base.sh
timeout-minutes: 60
- name: Deploy elizaOS to VM
working-directory: packages/os/linux
run: bash vm/scripts/deploy.sh
- name: Run VM integration tests
working-directory: packages/os/linux
timeout-minutes: 20
run: bash vm/scripts/run-tests.sh
- name: Generate VM bundle metadata
working-directory: packages/os/linux
run: |
mkdir -p vm/output
python3 vm/scripts/generate-bundle-metadata.py \
--output-dir vm/output/bundle-metadata
- name: Convert to OVA (for VirtualBox / VMware)
id: ova
if: inputs.format == 'ova' || inputs.format == 'all' || inputs.format == 'vmdk' || inputs.format == ''
run: |
set -euo pipefail
BASE_QCOW=$(find packages/os/linux -maxdepth 3 -name "*.qcow2" | head -1)
if [ -z "$BASE_QCOW" ]; then
echo "ERROR: no qcow2 image found after VM build — OVA conversion cannot proceed"
exit 1
fi
DATE=$(date +%Y.%m.%d)
VMDK="elizaos-vm-${DATE}.vmdk"
OVA="elizaos-vm-${DATE}.ova"
OVF="elizaos-vm-${DATE}.ovf"
echo "Converting qcow2 to vmdk..."
qemu-img convert -f qcow2 -O vmdk "$BASE_QCOW" "$VMDK"
# Minimal OVF descriptor
cat > "$OVF" << EOF
<?xml version="1.0" encoding="UTF-8"?>
<Envelope xmlns="http://schemas.dmtf.org/ovf/envelope/1"
xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1"
xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData">
<References>
<File ovf:href="${VMDK}" ovf:id="file1"/>
</References>
<VirtualSystem ovf:id="elizaOS">
<Name>elizaOS</Name>
<VirtualHardwareSection>
<Item>
<rasd:ElementName>4 virtual CPUs</rasd:ElementName>
<rasd:InstanceID>1</rasd:InstanceID>
<rasd:ResourceType>3</rasd:ResourceType>
<rasd:VirtualQuantity>4</rasd:VirtualQuantity>
</Item>
<Item>
<rasd:ElementName>4096 MB RAM</rasd:ElementName>
<rasd:InstanceID>2</rasd:InstanceID>
<rasd:ResourceType>4</rasd:ResourceType>
<rasd:VirtualQuantity>4096</rasd:VirtualQuantity>
</Item>
</VirtualHardwareSection>
</VirtualSystem>
</Envelope>
EOF
tar -cvf "$OVA" "$OVF" "$VMDK"
sha256sum "$OVA" > "${OVA}.sha256"
OVA_SHA256=$(cut -d' ' -f1 "${OVA}.sha256")
OVA_SIZE=$(stat --format="%s" "$OVA")
echo "ova_path=$OVA" >> "$GITHUB_OUTPUT"
echo "ova_sha256=$OVA_SHA256" >> "$GITHUB_OUTPUT"
echo "ova_size=$OVA_SIZE" >> "$GITHUB_OUTPUT"
echo "date=$DATE" >> "$GITHUB_OUTPUT"
- name: Attest SLSA build provenance for VM image
# Mints a Sigstore-signed in-toto/SLSA provenance attestation
# tied to this repo + workflow + commit, via GitHub OIDC.
# No long-lived secrets. Users verify with:
# gh attestation verify <ova-file> --owner elizaOS
# Only fires when an OVA was actually produced (the convert
# step is gated on the format input).
if: steps.ova.outputs.ova_path != ''
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373
with:
subject-path: ${{ steps.ova.outputs.ova_path }}
- name: Upload VM artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: elizaos-vm-image
path: |
elizaos-vm-*.ova
elizaos-vm-*.ova.sha256
elizaos-vm-*.qcow2
packages/os/linux/vm/output/
if-no-files-found: error
retention-days: 7
- name: Upload to GitHub Release
if: github.event_name == 'release' || (github.event_name == 'workflow_dispatch' && inputs.publish) || (github.event_name == 'workflow_call' && inputs.publish == true)
uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b
with:
files: |
elizaos-vm-*.ova
elizaos-vm-*.ova.sha256
fail_on_unmatched_files: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Write VM metadata to summary
run: |
{
echo "## elizaOS VM Image"
echo "| Field | Value |"
echo "| ----- | ----- |"
if [ -n "${{ steps.ova.outputs.ova_path }}" ]; then
echo "| OVA | \`${{ steps.ova.outputs.ova_path }}\` |"
echo "| SHA-256 | \`${{ steps.ova.outputs.ova_sha256 }}\` |"
echo "| Size | ${{ steps.ova.outputs.ova_size }} bytes |"
else
echo "| OVA | not produced |"
fi
echo "| Format | ${{ inputs.format || 'ova' }} |"
} >> "$GITHUB_STEP_SUMMARY"
+66
View File
@@ -0,0 +1,66 @@
# Cache-key stability gate
#
# Asserts that the Anthropic prompt-cache key inputs (Stage 1 / Stage 2 stable
# prefix hashes, STABLE_PLANNER_TOOLS envelope hash, HANDLE_RESPONSE +
# PLAN_ACTIONS tool envelope hashes) are byte-stable across PRs.
#
# If this job fails, the cached Anthropic prefix has been busted — every
# subsequent PR pays ~80% extra tokens until someone notices. Fix the source
# of the churn (typically a whitespace edit in a prompt template, a reordered
# action list, or an environment-dependent string that leaked into a stable
# prompt segment). Only rebase the baked-in hashes when the change is
# explicitly INTENTIONAL — see
# `docs/audits/lifeops-2026-05-11/cache-key-stability.md`.
#
# Local equivalent (run before pushing):
#
# bun run test:cache-stability
name: Cache-key Stability
on:
pull_request:
branches: [main]
paths:
- "packages/core/**"
- "plugins/plugin-personal-assistant/**"
- "packages/prompts/**"
- ".github/workflows/cache-key-stability.yml"
concurrency:
group: cache-key-stability-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
BUN_VERSION: "canary"
NODE_VERSION: "24.15.0"
jobs:
cache-key-stability:
name: cache-key-stability
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 10
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
fetch-depth: 1
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install --ignore-scripts --no-frozen-lockfile
install-native-deps: "false"
- name: Run cache-key stability test
run: bun run test:cache-stability
+206
View File
@@ -0,0 +1,206 @@
# Cerebras Nightly — continuous benchmark grind (Phase 8)
#
# Runs the lifeops + personality benchmark suites nightly against
# eliza / hermes / openclaw with Cerebras gemma-4-31b as both agent
# (where applicable) and judge. Produces a delta report vs the previous
# nightly's results so drift surfaces without anyone having to look.
#
# This is the workflow `Phase 8 — Continuous Cerebras grind in CI` from
# the cleanup roadmap. Goal: free judge calls, daily cadence, drift
# visibility.
#
# Required secret (job skipped, not failed, when missing):
# CEREBRAS_API_KEY — gemma-4-31b agent + judge
#
# Optional secrets:
# GITHUB_TOKEN — present by default; required to upsert the
# tracking issue comment.
name: Cerebras Nightly
on:
schedule:
# 03:00 UTC daily — Mockoon dependencies are quiet then, and the
# ~$10 daily cost lands on the right billing cycle.
- cron: "0 3 * * *"
workflow_dispatch:
inputs:
suite:
description: "Benchmark suite (lifeops | personality | both)"
required: false
type: choice
options: [both, lifeops, personality]
default: both
limit:
description: "Scenarios per agent per domain (default 10)"
required: false
type: string
default: "10"
tracking_issue:
description: "Tracking issue number for delta comments (optional)"
required: false
type: string
default: ""
concurrency:
group: cerebras-nightly-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: read
issues: write
actions: read
env:
BUN_VERSION: "canary"
NODE_VERSION: "24.15.0"
PYTHON_VERSION: "3.12"
jobs:
bench:
name: "nightly: ${{ matrix.suite }}"
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
suite: ["lifeops", "personality"]
steps:
- name: Check secrets
id: gate
env:
CEREBRAS_API_KEY: ${{ secrets.CEREBRAS_API_KEY }}
run: |
if [ -z "${CEREBRAS_API_KEY:-}" ]; then
echo "[cerebras-nightly] CEREBRAS_API_KEY not configured — skipping."
echo "skip=true" >> "$GITHUB_OUTPUT"
else
# Honor the workflow_dispatch suite filter.
requested="${{ github.event.inputs.suite || 'both' }}"
if [ "$requested" != "both" ] && [ "$requested" != "${{ matrix.suite }}" ]; then
echo "[cerebras-nightly] suite ${{ matrix.suite }} not requested ($requested) — skipping."
echo "skip=true" >> "$GITHUB_OUTPUT"
else
echo "skip=false" >> "$GITHUB_OUTPUT"
fi
fi
- name: Checkout
if: steps.gate.outputs.skip != 'true'
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
fetch-depth: 1
- name: Setup Node.js
if: steps.gate.outputs.skip != 'true'
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Python
if: steps.gate.outputs.skip != 'true'
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1
with:
python-version: ${{ env.PYTHON_VERSION }}
- name: Setup workspace
if: steps.gate.outputs.skip != 'true'
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install --ignore-scripts --no-frozen-lockfile
- name: Install Python deps
if: steps.gate.outputs.skip != 'true'
run: |
python -m pip install --upgrade pip
# benchmark suites live in packages/benchmarks; install editable
# for each adapter we touch.
for pkg in lifeops-bench eliza-adapter hermes-adapter openclaw-adapter personality-bench; do
if [ -d "packages/benchmarks/$pkg" ]; then
python -m pip install -e "packages/benchmarks/$pkg" || true
fi
done
- name: Run nightly bench
if: steps.gate.outputs.skip != 'true'
id: bench
env:
CEREBRAS_API_KEY: ${{ secrets.CEREBRAS_API_KEY }}
CEREBRAS_BASE_URL: "https://api.cerebras.ai/v1"
CEREBRAS_MODEL: "gemma-4-31b"
EVAL_MODEL_PROVIDER: "cerebras"
NIGHTLY_LIMIT: "${{ github.event.inputs.limit || '10' }}"
RUN_ID: "cerebras-nightly-${{ github.run_id }}"
run: |
set -euo pipefail
mkdir -p artifacts/${{ matrix.suite }}
if [ "${{ matrix.suite }}" = "lifeops" ]; then
for agent in eliza hermes openclaw; do
echo "::group::lifeops/$agent"
python -m eliza_lifeops_bench \
--agent "$agent" \
--suite full \
--limit "$NIGHTLY_LIMIT" \
--concurrency 2 \
--output "artifacts/lifeops/$agent" \
|| echo "[lifeops/$agent] non-zero exit; continuing"
echo "::endgroup::"
done
elif [ "${{ matrix.suite }}" = "personality" ]; then
for agent in eliza hermes openclaw eliza-runtime; do
echo "::group::personality/$agent"
node scripts/personality-bench-run.mjs \
--agent "$agent" \
--limit "$NIGHTLY_LIMIT" \
--output "artifacts/personality/$agent" \
|| echo "[personality/$agent] non-zero exit; continuing"
echo "::endgroup::"
done
fi
- name: Generate delta report
if: steps.gate.outputs.skip != 'true'
id: delta
run: |
set -euo pipefail
# Compare against the previous nightly's results if available.
# Stored under .github/cerebras-nightly-baseline/<suite>.json by
# the prior run via cache, or fetched from artifacts when cache
# misses. Best-effort — on miss we report absolute scores.
python scripts/cerebras-nightly-delta.py \
--suite "${{ matrix.suite }}" \
--input artifacts/${{ matrix.suite }} \
--output artifacts/${{ matrix.suite }}/delta.md \
--json-output artifacts/${{ matrix.suite }}/delta.json \
|| true
- name: Upload artifacts
if: always() && steps.gate.outputs.skip != 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: cerebras-nightly-${{ matrix.suite }}-${{ github.run_id }}
path: |
artifacts/${{ matrix.suite }}/
retention-days: 90
if-no-files-found: warn
- name: Upsert tracking issue comment
if: steps.gate.outputs.skip != 'true' && github.event.inputs.tracking_issue != ''
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
ISSUE: ${{ github.event.inputs.tracking_issue }}
SUITE: ${{ matrix.suite }}
run: |
set -euo pipefail
if [ -f "artifacts/$SUITE/delta.md" ]; then
{
printf '## Cerebras Nightly — %s — run %s\n\n' "$SUITE" "${{ github.run_id }}"
cat "artifacts/$SUITE/delta.md"
printf '\n[Full artifacts](https://github.com/%s/actions/runs/%s)\n' "${{ github.repository }}" "${{ github.run_id }}"
} > /tmp/cerebras-nightly-comment.md
gh issue comment "$ISSUE" --body-file /tmp/cerebras-nightly-comment.md \
|| echo "tracking-issue upsert failed; artifact retained"
fi
+108
View File
@@ -0,0 +1,108 @@
# certification-image — builds and publishes the GPU certification image
# (docker/certification/Dockerfile.gpu) that vast.ai certification runs boot
# (#14548, epic #14541). The image bakes everything the 16 KB onstart script
# cannot carry: CUDA llama-server, the sha256-pinned gpu-vision models,
# node24/bun/playwright/tesseract/ffmpeg. Pushes
# ghcr.io/elizaos/certification-gpu:{latest,sha-<short>} via GITHUB_TOKEN —
# no extra registry secret. Rebuilds are rare (image inputs only), so this
# triggers on dispatch plus pushes that touch its own inputs, never on PRs.
name: certification-image
on:
workflow_dispatch:
push:
branches: [develop]
paths:
- "docker/certification/Dockerfile.gpu"
- "scripts/gpu-vision/setup.mjs"
- "scripts/gpu-vision/lib.mjs"
- "scripts/gpu-vision/models.lock.json"
- ".github/workflows/certification-image.yml"
concurrency:
group: certification-image-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: read
env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository_owner }}/certification-gpu
jobs:
build-and-push:
name: build-and-push
# Same fleet fallback as build-agent-image: hetzner when online (the
# image is ~15 GB with baked models and CUDA layers; hosted runners need
# the disk-free step below to survive it).
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 180
permissions:
contents: read
packages: write
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Free up disk space
run: |
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL || true
sudo docker image prune -a -f || true
df -h /
- uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c
with:
driver-opts: |
env.BUILDKIT_STEP_LOG_MAX_SIZE=10485760
env.BUILDKIT_STEP_LOG_MAX_SPEED=10485760
# Docker tags must be lowercase; github.repository_owner is "elizaOS".
- id: image
run: echo "name=${REGISTRY}/${IMAGE_NAME,,}" >> "$GITHUB_OUTPUT"
- id: meta
uses: docker/metadata-action@80c7e94dd9b9319bd5eb7a0e0fe9291e23a2a2e9
with:
images: ${{ steps.image.outputs.name }}
# :latest tracks develop — it is what run-certification.mjs boots by
# default; :sha-<short> pins for reproducible re-runs.
tags: |
type=raw,value=latest,enable=${{ github.ref == 'refs/heads/develop' }}
type=sha,prefix=sha-,format=short
- name: Build and push
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a
with:
context: .
file: docker/certification/Dockerfile.gpu
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# Single-manifest image: vast pulls by tag, provenance attestation
# manifests confuse some of its pullers.
provenance: false
cache-from: type=gha
cache-to: type=gha,mode=max
- name: Image summary
run: |
{
echo "## certification-image"
echo ""
echo "Pushed tags:"
echo '```'
echo "${{ steps.meta.outputs.tags }}"
echo '```'
echo ""
echo "Boot it on vast: \`node scripts/vast/run-certification.mjs --image ${{ steps.image.outputs.name }}:latest ...\`"
} >> "$GITHUB_STEP_SUMMARY"
+199
View File
@@ -0,0 +1,199 @@
# certification-vast — dispatches an automated full-tier certification run on
# a rented vast.ai GPU (#14548, epic #14541). Thin caller of
# scripts/vast/run-certification.mjs: search offers → create from the
# prebuilt certification image → onstart runs the packages/evidence chain
# (bundle:create → certify:rollup → certify:sign) → poll → pull the signed
# certification.json → DESTROY the instance in the script's own finally.
#
# Triggers: workflow_dispatch (any certifier with repo write), plus a nightly
# cron that is a no-op unless the repo variable ELIZA_VAST_CERT_ENABLED is
# the string 'true' — cost is real money, so nightly spend is owner-opt-in.
# Deliberately NO pull_request trigger: this job holds VAST_API_KEY and the
# ELIZA_CERT_SIGNING_KEY; PR-triggered code must never see either.
#
# Secrets discipline: both secrets are exposed only to the single step that
# runs the script, and the script forwards the signing key solely inside the
# create-instance env payload (never the onstart text, never logs). When vast
# fails for ANY reason the job goes red with a "local certification required"
# summary carrying the exact fallback command — same chain, same output; the
# develop→main gate (certification-verify.yml) cannot tell the difference.
name: certification-vast
on:
workflow_dispatch:
inputs:
sha:
description: "Commit sha to certify (default: the ref this workflow runs on)"
required: false
type: string
tier:
description: "Certification tier"
required: false
default: "full"
type: choice
options: [cpu, gpu, full]
max_dph:
description: "Budget cap in $/hr per offer"
required: false
default: "0.60"
type: string
max_attempts:
description: "Max offers tried before giving up"
required: false
default: "3"
type: string
timeout_minutes:
description: "Hard wall-clock cap per attempt"
required: false
default: "120"
type: string
schedule:
# Nightly on develop, 06:17 UTC (off the top-of-hour crunch); gated below
# on vars.ELIZA_VAST_CERT_ENABLED == 'true'.
- cron: "17 6 * * *"
concurrency:
group: certification-vast
cancel-in-progress: false
permissions:
contents: read
jobs:
vast-certification:
name: vast-certification
if: github.event_name == 'workflow_dispatch' || vars.ELIZA_VAST_CERT_ENABLED == 'true'
runs-on: ubuntu-latest
timeout-minutes: 400
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Resolve certified sha
id: sha
env:
INPUT_SHA: ${{ inputs.sha }}
run: |
set -euo pipefail
sha="${INPUT_SHA:-$GITHUB_SHA}"
if ! [[ "$sha" =~ ^[0-9a-fA-F]{7,40}$ ]]; then
echo "::error title=certification-vast::input sha '$sha' is not a git sha"
exit 1
fi
echo "sha=$sha" >> "$GITHUB_OUTPUT"
# Fail in seconds — before any vast spend — when the run cannot possibly
# produce a verifiable certification.
- name: Preflight secrets
env:
HAS_VAST_KEY: ${{ secrets.VAST_API_KEY != '' }}
HAS_SIGNING_KEY: ${{ secrets.ELIZA_CERT_SIGNING_KEY != '' }}
run: |
set -euo pipefail
missing=""
[ "$HAS_VAST_KEY" = "true" ] || missing="$missing VAST_API_KEY"
[ "$HAS_SIGNING_KEY" = "true" ] || missing="$missing ELIZA_CERT_SIGNING_KEY"
if [ -n "$missing" ]; then
echo "::error title=certification-vast::missing repo secret(s):$missing"
exit 1
fi
# Node 24 to match the script's target runtime (plain node, zero
# workspace deps — no bun install on this runner).
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24"
- name: Dry-run plan (no API calls, no secrets in output)
env:
CERT_SHA: ${{ steps.sha.outputs.sha }}
TIER: ${{ inputs.tier || 'full' }}
MAX_DPH: ${{ inputs.max_dph || '0.60' }}
MAX_ATTEMPTS: ${{ inputs.max_attempts || '3' }}
TIMEOUT_MINUTES: ${{ inputs.timeout_minutes || '120' }}
run: |
set -euo pipefail
node scripts/vast/run-certification.mjs \
--sha "$CERT_SHA" --tier "$TIER" \
--max-dph "$MAX_DPH" --max-attempts "$MAX_ATTEMPTS" \
--timeout-minutes "$TIMEOUT_MINUTES" \
--dry-run
# The ONLY step with secret access. The signing key travels to the
# instance exclusively inside the create-payload env; the vast key never
# leaves this runner. CERT_PUSH_CMD is optional owner-configured storage
# push (R2/S3) — without it the signed certification still comes back
# via the instance logs.
- name: Run certification on vast.ai
id: run
env:
VAST_API_KEY: ${{ secrets.VAST_API_KEY }}
ELIZA_CERT_SIGNING_KEY: ${{ secrets.ELIZA_CERT_SIGNING_KEY }}
CERT_PUSH_CMD: ${{ secrets.ELIZA_CERT_PUSH_CMD }}
CERT_SHA: ${{ steps.sha.outputs.sha }}
TIER: ${{ inputs.tier || 'full' }}
MAX_DPH: ${{ inputs.max_dph || '0.60' }}
MAX_ATTEMPTS: ${{ inputs.max_attempts || '3' }}
TIMEOUT_MINUTES: ${{ inputs.timeout_minutes || '120' }}
run: |
set -euo pipefail
node scripts/vast/run-certification.mjs \
--sha "$CERT_SHA" --tier "$TIER" \
--max-dph "$MAX_DPH" --max-attempts "$MAX_ATTEMPTS" \
--timeout-minutes "$TIMEOUT_MINUTES" \
--out vast-certification-output
# The signed certification + instance logs are the run's product; the
# promotion PR author downloads this artifact and commits
# certification.json (+ the pushed bundle) per the gate's runbook.
- name: Upload certification artifact
if: ${{ !cancelled() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: vast-certification-${{ steps.sha.outputs.sha }}
path: vast-certification-output/
if-no-files-found: ignore
retention-days: 14
- name: Success summary
if: success()
env:
CERT_SHA: ${{ steps.sha.outputs.sha }}
run: |
{
echo "## certification-vast ✅"
echo ""
echo "Signed certification for \`$CERT_SHA\` pulled from the vast instance"
echo "(artifact **vast-certification-$CERT_SHA**, cost logged in the run step)."
echo ""
echo "To use it on a promotion PR: download the artifact, commit"
echo "\`certification.json\` at the repo root and the bundle at \`evidence/bundle/\`"
echo "per [.github/certification/README.md](https://github.com/${GITHUB_REPOSITORY}/blob/develop/.github/certification/README.md)."
} >> "$GITHUB_STEP_SUMMARY"
# The documented fallback: same chain, same output, run by a human/agent
# holding the signing key. Kept in lockstep with scripts/vast/README.md.
- name: Local certification required (vast run failed)
if: failure()
env:
CERT_SHA: ${{ steps.sha.outputs.sha }}
TIER: ${{ inputs.tier || 'full' }}
run: |
{
echo "## certification-vast ❌ — local certification required"
echo ""
echo "The vast.ai run for \`$CERT_SHA\` failed (dead key / no offers / stuck instance /"
echo "onstart failure — exit code in the run step above; every path attempts instance"
echo "destroy first). Certify locally on a machine holding \`ELIZA_CERT_SIGNING_KEY\`:"
echo ""
echo '```bash'
echo "git fetch origin $CERT_SHA && git checkout $CERT_SHA"
echo "node scripts/vast/local-certify.mjs --tier $TIER"
echo '```'
echo ""
echo "Runbook (costs, kill paths, storage push): [scripts/vast/README.md](https://github.com/${GITHUB_REPOSITORY}/blob/develop/scripts/vast/README.md)"
} >> "$GITHUB_STEP_SUMMARY"
echo "::error title=certification-vast::vast run failed — local certification required: node scripts/vast/local-certify.mjs --tier $TIER (see scripts/vast/README.md)"
+265
View File
@@ -0,0 +1,265 @@
# certification-verify — the develop→main promotion gate (#14547, epic #14541).
#
# PRs targeting `main` (a production deploy) must carry a `certification.json`
# at the repo root: an Ed25519-signed claim (produced by
# `packages/evidence certify:sign`, protocol in #14546) that a trusted
# keyholder reviewed the evidence bundle for the promoted commit. This
# workflow is a THIN CALLER of `certify:verify` — zero verification logic
# lives here — plus a docs/template-only commit-drift check
# (scripts/certification/check-commit-drift.mjs) and a human-readable check
# summary showing WHO certified WHAT.
#
# Required-check name for main's branch protection: `verify-certification`
# (admin wiring + break-glass runbook: .github/certification/README.md).
#
# The `certification-verify-selftest` job runs on PRs (to main or develop)
# that touch the gate's own plumbing: it generates a throwaway keypair, signs
# a fixture bundle with the real CLIs, and asserts green-verifies + tampered
# fails — so gate changes are CI-tested without a real certification. It is
# NOT the required check; skipping its steps on unrelated PRs is safe.
name: certification-verify
on:
pull_request:
# `main` for the gate itself; `develop` only so the selftest job covers
# PRs that change the gate's plumbing (which target develop).
branches: [main, develop]
types: [opened, synchronize, reopened, ready_for_review]
concurrency:
group: certification-verify-${{ github.event.pull_request.number }}
cancel-in-progress: true
permissions:
contents: read
jobs:
verify-certification:
name: verify-certification
# Draft promotion PRs skip the gate: GitHub cannot merge a draft, and the
# ready_for_review trigger re-runs the check the moment it matters.
if: github.event.pull_request.base.ref == 'main' && github.event.pull_request.draft == false
runs-on: ubuntu-latest
timeout-minutes: 20
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
HEAD_SHA: ${{ github.event.pull_request.head.sha }}
ELIZA_SKIP_ARTIFACT_SYNC: "1"
steps:
- name: Checkout PR head
# actions/checkout@v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
# The exact head commit, not the synthetic merge ref: the
# certification binds to a real commit sha, and the drift check
# compares that sha against this head.
ref: ${{ github.event.pull_request.head.sha }}
# Full history so `git diff cert.commit..head` and the ancestry
# check resolve however far back the certified commit sits.
fetch-depth: 0
# TRUST ANCHOR — read the public key from the BASE branch (main), never
# from the PR head. If this step read the PR's own copy of the key, an
# attacker could swap the key and a matching self-signed
# certification.json in a single promotion PR and the gate would happily
# verify their own signature. Reading main's copy means a key only
# becomes trusted after it lands on main through review; until then a
# foreign-key certification fails `wrong-key`. (Rotation + bootstrap:
# .github/certification/README.md.)
- name: Read trusted public key from the base branch
run: |
set -euo pipefail
git fetch --quiet --no-tags origin "+refs/heads/${BASE_REF}:refs/remotes/origin/${BASE_REF}"
if ! git show "refs/remotes/origin/${BASE_REF}:.github/certification/certification-public-key.pem" \
> "$RUNNER_TEMP/trusted-public-key.pem"; then
echo "::error title=certification-verify::No trusted public key at .github/certification/certification-public-key.pem on ${BASE_REF}. Until the certification stack lands on ${BASE_REF}, this gate cannot pass (bootstrap: see .github/certification/README.md)."
exit 1
fi
# Artifact convention: `certification.json` committed at the repo root,
# with the signed evidence bundle committed at `evidence/bundle/`.
# Checked before dependency install so missing artifacts fail in seconds,
# with remediation.
- name: Locate certification and evidence bundle
run: |
set -euo pipefail
if [ ! -f certification.json ]; then
{
echo "## certification-verify"
echo ""
echo "**Result:** ❌ no \`certification.json\` at the repo root of this promotion branch."
echo ""
echo "Produce one by running the vast certification workflow for this commit, or locally:"
echo ""
echo '```bash'
echo "bun run --cwd packages/evidence bundle:create -- --tier full"
echo "bun run --cwd packages/evidence certify:rollup -- --bundle <bundle-dir> --out verdicts.json"
echo "# review + edit verdicts.json, then (requires the ELIZA_CERT_SIGNING_KEY secret):"
echo "bun run --cwd packages/evidence certify:sign -- --bundle <bundle-dir> --verdicts verdicts.json --reviewer-id <you> --reviewer-kind human"
echo "cp <bundle-dir>/certification.json ./certification.json"
echo "rm -rf evidence/bundle && mkdir -p evidence && cp -R <bundle-dir> evidence/bundle"
echo '```'
echo ""
echo "Full runbook: [.github/certification/README.md](https://github.com/${GITHUB_REPOSITORY}/blob/${BASE_REF}/.github/certification/README.md)"
} >> "$GITHUB_STEP_SUMMARY"
echo "::error title=certification-verify::certification.json missing at the repo root — run the vast certification workflow or local certify:sign; see .github/certification/README.md"
exit 1
fi
if [ ! -f evidence/bundle/manifest.json ]; then
{
echo "## certification-verify"
echo ""
echo "**Result:** ❌ no evidence bundle at \`evidence/bundle/\` on this promotion branch."
echo ""
echo "The promotion gate always passes \`--bundle evidence/bundle\` to \`certify:verify\` so artifact integrity, rollup completeness, and signed evidence-path membership are re-derived from the committed bundle."
echo ""
echo '```bash'
echo "bun run --cwd packages/evidence bundle:create -- --tier full"
echo "bun run --cwd packages/evidence certify:rollup -- --bundle <bundle-dir> --out verdicts.json"
echo "bun run --cwd packages/evidence certify:sign -- --bundle <bundle-dir> --verdicts verdicts.json --reviewer-id <you> --reviewer-kind human"
echo "cp <bundle-dir>/certification.json ./certification.json"
echo "rm -rf evidence/bundle && mkdir -p evidence && cp -R <bundle-dir> evidence/bundle"
echo '```'
echo ""
echo "Full runbook: [.github/certification/README.md](https://github.com/${GITHUB_REPOSITORY}/blob/${BASE_REF}/.github/certification/README.md)"
} >> "$GITHUB_STEP_SUMMARY"
echo "::error title=certification-verify::evidence/bundle/manifest.json missing — commit the signed evidence bundle and re-run the gate"
exit 1
fi
echo "certification.json present"
echo "evidence/bundle present"
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: "1.3.14" # pinned: floating canary rewrites bun.lock (#11184/#9454)
install-native-deps: "false"
install-protoc: "false"
setup-python: "false"
run-postinstall: "false" # certify:verify needs only zod; skips the multi-GB artifact sync
skip-avatar-clone: "true"
no-vision-deps: "true"
# The certification may sit a few commits behind head when only docs
# moved after signing. The helper passes iff head == cert.commit, or
# cert.commit is an ancestor and every drifted path is docs/template-only
# (docs/**, packages/docs/**, *.md, plus signed certification artifacts).
# Any source, workflow, policy, or config drift forces re-certification.
- name: Check commit drift (certification commit vs PR head)
id: drift
run: |
set -euo pipefail
status=0
node scripts/certification/check-commit-drift.mjs \
--cert "$GITHUB_WORKSPACE/certification.json" \
--head "$HEAD_SHA" \
--repo "$GITHUB_WORKSPACE" \
--json-out "$RUNNER_TEMP/drift.json" \
--github-output "$GITHUB_OUTPUT" || status=$?
echo "status=$status" >> "$GITHUB_OUTPUT"
- name: Verify certification (certify:verify)
id: verify
env:
CERT_COMMIT: ${{ steps.drift.outputs.cert-commit }}
run: |
set -euo pipefail
args=(
--cert "$GITHUB_WORKSPACE/certification.json"
--pubkey "$RUNNER_TEMP/trusted-public-key.pem"
--max-age-hours 72
--required-tier full
--json
)
# The commit binding: certify:verify checks the SIGNED commit; the
# drift step above already bound that commit to this PR head.
if [ -n "$CERT_COMMIT" ]; then
args+=( --expected-commit "$CERT_COMMIT" )
fi
# Required committed bundle: verify re-hashes every artifact and
# re-derives the mechanical rollup and evidence-path membership
# (bundle-tampered / verdict-incomplete).
args+=( --bundle "$GITHUB_WORKSPACE/evidence/bundle" )
status=0
bun run --cwd packages/evidence certify:verify -- "${args[@]}" \
> "$RUNNER_TEMP/certification-report.json" || status=$?
echo "status=$status" >> "$GITHUB_OUTPUT"
- name: Render check summary
if: ${{ !cancelled() }}
run: |
set -euo pipefail
node scripts/certification/render-check-summary.mjs \
--mode summary \
--report "$RUNNER_TEMP/certification-report.json" \
--drift "$RUNNER_TEMP/drift.json" \
--pubkey "$RUNNER_TEMP/trusted-public-key.pem" \
--cert-path "certification.json (repo root, head ${HEAD_SHA})" \
>> "$GITHUB_STEP_SUMMARY"
- name: Enforce gate result
env:
DRIFT_STATUS: ${{ steps.drift.outputs.status }}
VERIFY_STATUS: ${{ steps.verify.outputs.status }}
run: |
set -euo pipefail
node scripts/certification/render-check-summary.mjs \
--mode annotations \
--report "$RUNNER_TEMP/certification-report.json" \
--drift "$RUNNER_TEMP/drift.json"
if [ "$DRIFT_STATUS" != "0" ] || [ "$VERIFY_STATUS" != "0" ]; then
echo "::error title=certification-verify::gate failed (commit-drift status=$DRIFT_STATUS, certify:verify status=$VERIFY_STATUS) — failure codes above and in the step summary"
exit 1
fi
echo "certification verified"
certification-verify-selftest:
name: certification-verify-selftest
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
contents: read
pull-requests: read
steps:
- name: Detect gate-relevant changes
id: changes
env:
GH_TOKEN: ${{ github.token }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: |
set -euo pipefail
files="$(gh api "repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}/files" --paginate --jq '.[].filename')"
if printf '%s\n' "$files" | grep -Eq '^(\.github/workflows/certification-verify\.yml$|scripts/certification/|packages/evidence/|\.github/certification/)'; then
echo "relevant=true" >> "$GITHUB_OUTPUT"
else
echo "relevant=false" >> "$GITHUB_OUTPUT"
echo "gate plumbing untouched by this PR; selftest not needed"
fi
- name: Checkout
if: steps.changes.outputs.relevant == 'true'
# actions/checkout@v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup workspace dependencies
if: steps.changes.outputs.relevant == 'true'
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: "1.3.14" # pinned: floating canary rewrites bun.lock (#11184/#9454)
install-native-deps: "false"
install-protoc: "false"
setup-python: "false"
run-postinstall: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
env:
ELIZA_SKIP_ARTIFACT_SYNC: "1"
- name: Commit-drift helper tests
if: steps.changes.outputs.relevant == 'true'
run: node --test scripts/certification/check-commit-drift.test.mjs
- name: Gate plumbing selftest (keygen → sign → verify → tamper)
if: steps.changes.outputs.relevant == 'true'
run: node scripts/certification/selftest.mjs
+252
View File
@@ -0,0 +1,252 @@
name: Chat Latency Live
# Manual, privacy-safe comparison of Cerebras direct and the Eliza Cloud model
# gateway from the same GitHub-hosted runner. Production is never automatic:
# selecting it binds the job to the protected production environment.
on:
pull_request:
branches: [develop, main]
paths:
- ".github/workflows/chat-latency-live.yml"
- "packages/scripts/cloud/chat-latency.mjs"
- "packages/scripts/cloud/chat-latency.test.mjs"
- "packages/scripts/__tests__/chat-latency-workflow.test.ts"
- "package.json"
workflow_dispatch:
inputs:
environment:
description: Eliza Cloud gateway environment
required: true
default: staging
type: choice
options:
- staging
- production
repeats:
description: Repetitions for each model/reasoning case
required: true
default: "30"
type: choice
options:
- "3"
- "10"
- "30"
expected_gateway_sha:
description: Exact 40-character commit expected from /api/health
required: true
type: string
concurrency:
group: chat-latency-live-${{ github.event_name }}-${{ inputs.environment || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
permissions:
contents: read
jobs:
contract:
name: Probe contract
runs-on: ubuntu-24.04
timeout-minutes: 5
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24.15.0"
- name: Validate probe syntax and privacy contract
run: |
node --check packages/scripts/cloud/chat-latency.mjs
node --test packages/scripts/cloud/chat-latency.test.mjs
live:
name: paired direct + gateway (${{ inputs.environment }})
if: github.event_name == 'workflow_dispatch'
needs: contract
runs-on: ubuntu-24.04
timeout-minutes: 90
environment: ${{ inputs.environment }}
env:
ELIZA_CLOUD_CHAT_LATENCY_BASE_URL: ${{ inputs.environment == 'production' && 'https://api.elizacloud.ai' || 'https://api-staging.elizacloud.ai' }}
ELIZA_GATEWAY_DEPLOY_SHA: ${{ inputs.expected_gateway_sha }}
REPORT_PATH: reports/chat-latency-paired-${{ inputs.environment }}.jsonl
steps:
- name: Checkout exact requested ref
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24.15.0"
- name: Verify the exact deployed gateway SHA
env:
EXPECTED_GATEWAY_SHA: ${{ inputs.expected_gateway_sha }}
run: |
node --input-type=module - <<'NODE'
const expected = process.env.EXPECTED_GATEWAY_SHA ?? "";
const base = process.env.ELIZA_CLOUD_CHAT_LATENCY_BASE_URL ?? "";
if (!/^[a-f0-9]{40}$/.test(expected)) {
throw new Error("expected_gateway_sha must be a 40-character lowercase commit");
}
const response = await fetch(base + "/api/health", {
headers: { "user-agent": "eliza-chat-latency/1.0" },
signal: AbortSignal.timeout(30_000),
});
if (!response.ok) {
throw new Error("Gateway health returned HTTP " + response.status);
}
const body = await response.json();
if (body?.commit !== expected) {
throw new Error(
"Gateway serves " + (body?.commit ?? "<missing>") + ", expected " + expected,
);
}
console.log(
"Verified gateway deployment " +
expected +
" (" +
(body.environment ?? "unknown") +
")",
);
NODE
- name: Probe Gemma and GLM reasoning modes on one runner
id: probe
shell: bash
env:
CEREBRAS_CHAT_LATENCY_API_KEY: ${{ secrets.CEREBRAS_API_KEY }}
ELIZA_CLOUD_CHAT_LATENCY_API_KEY: ${{ secrets.ELIZACLOUD_API_KEY }}
ELIZA_GATEWAY_DEPLOY_SHA: ${{ inputs.expected_gateway_sha }}
BENCHMARK_SEED: ${{ github.run_id }}-${{ github.run_attempt }}
run: |
mkdir -p reports
set +e
# One process holds runner, clock, prompts, and case set constant.
# It randomizes case order, alternates target-first order exactly for
# each case and labels cold/warm/post-idle phases.
# It idles before every target labeled post-idle.
# Warm pairs are paced at 40 gateway requests/minute, below the
# lowest Cloud organization tier, so this remains a latency test.
node packages/scripts/cloud/chat-latency.mjs \
--target paired \
--direct-base-url https://api.cerebras.ai \
--gateway-base-url "$ELIZA_CLOUD_CHAT_LATENCY_BASE_URL" \
--direct-api-key-env CEREBRAS_CHAT_LATENCY_API_KEY \
--gateway-api-key-env ELIZA_CLOUD_CHAT_LATENCY_API_KEY \
--repeat "${{ inputs.repeats }}" \
--timeout-ms 30000 \
--idle-ms 30000 \
--pair-interval-ms 1500 \
--max-proof-miss-rate 0.05 \
--seed "$BENCHMARK_SEED" \
--case gemma-4-31b@omit@512 \
--case gemma-4-31b@none@512 \
--case zai-glm-4.7@omit@4096 \
--case zai-glm-4.7@none@4096 \
--case zai-glm-4.7@none@512 \
| tee "$REPORT_PATH"
probe_status="${PIPESTATUS[0]}"
set -e
echo "status=$probe_status" >> "$GITHUB_OUTPUT"
- name: Reverify the exact deployed gateway SHA
if: always()
env:
EXPECTED_GATEWAY_SHA: ${{ inputs.expected_gateway_sha }}
run: |
node --input-type=module - <<'NODE'
const expected = process.env.EXPECTED_GATEWAY_SHA ?? "";
const base = process.env.ELIZA_CLOUD_CHAT_LATENCY_BASE_URL ?? "";
const response = await fetch(base + "/api/health", {
headers: { "user-agent": "eliza-chat-latency/1.0" },
signal: AbortSignal.timeout(30_000),
});
if (!response.ok) {
throw new Error("Gateway health returned HTTP " + response.status);
}
const body = await response.json();
if (body?.commit !== expected) {
throw new Error(
"Gateway changed during the benchmark: served " +
(body?.commit ?? "<missing>") +
", expected " +
expected,
);
}
console.log("Gateway deployment remained exact through benchmark completion");
NODE
- name: Add privacy-safe timing table to summary
if: always() && hashFiles(env.REPORT_PATH) != ''
run: |
node --input-type=module - "$REPORT_PATH" >> "$GITHUB_STEP_SUMMARY" <<'NODE'
import { readFileSync } from "node:fs";
import { summarizeLatencyRecords } from "./packages/scripts/cloud/chat-latency.mjs";
const path = process.argv[2];
const records = readFileSync(path, "utf8")
.split(/\r?\n/)
.filter(Boolean)
.map((line) => JSON.parse(line));
const summaries = summarizeLatencyRecords(records);
console.log("### Paired chat latency: direct + gateway / ${{ inputs.environment }}");
console.log("");
console.log(
"Gateway deployment: " +
(process.env.ELIZA_GATEWAY_DEPLOY_SHA ?? "unknown"),
);
console.log("");
console.log("| target | model | reasoning | cap | ok/total | TTFT p50/p90/p95 ms | total p50/p90/p95 ms | preforward p50/p90/p95 ms |");
console.log("| --- | --- | --- | ---: | ---: | ---: | ---: | ---: |");
const fmt = (metric) =>
[metric.p50, metric.p90, metric.p95]
.map((value) => value ?? "—")
.join(" / ");
for (const row of summaries) {
const values = [
row.target,
row.model,
row.reasoningEffort,
row.maxTokens,
row.successes + "/" + row.samples,
fmt(row.firstTokenMs),
fmt(row.totalMs),
fmt(row.preforwardMs),
];
console.log("| " + values.join(" | ") + " |");
}
const phaseCounts = Object.groupBy(
records,
(row) => row.phase ?? "unlabelled",
);
console.log("");
console.log(
"Phases: " +
Object.entries(phaseCounts)
.map(([phase, rows]) => phase + "=" + rows.length)
.join(", "),
);
NODE
- name: Upload exact-SHA latency evidence
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: chat-latency-paired-${{ inputs.environment }}-${{ inputs.expected_gateway_sha }}
path: ${{ env.REPORT_PATH }}
if-no-files-found: error
retention-days: 14
- name: Enforce probe result
if: always()
run: |
status="${{ steps.probe.outputs.status }}"
if [ -z "$status" ]; then
echo "::error::Probe did not produce a status."
exit 1
fi
exit "$status"
+232
View File
@@ -0,0 +1,232 @@
name: Chat shell gestures
# Real-browser gesture + interleaving e2e for the floating chat shell
# (packages/ui ContinuousChatOverlay + home screen), the component-tree
# render-parity contract, and the CHAT_GESTURE_COVERAGE gate (#12188).
#
# All 11 __e2e__ runners compose the shared runner toolkit
# (packages/ui/src/testing/e2e-runner) and record a stable-named .webm. This lane
# drives the REAL surfaces with real pointer/touch gestures in headless chromium,
# runs the boot-free coverage gate that fails when a gesture-handler site lacks a
# matrix row, and uploads every runner's video + screenshots.
#
# One shell runner is intentionally NOT in this lane:
# - run-fused-wake-integration-e2e drives the REAL native desktop wake producer
# (libwakeword + the wakeword-cpp build), which this lane does not compile
# (install-native-deps:false) — it self-skips without the lib, so it lives in
# the native-desktop build lane.
#
# run-perf-gate-e2e USED to be excluded here for a CLS 0.80 failure, but that
# shift was the conversation-swipe layout thrash: #13531 removed swipe (one
# infinite thread) and #13826 retargeted the gate onto scroll + maximize/restore,
# so CLS is now 0.0000 on develop (stable across runs, not timing-flaky). It is
# re-enabled below (#14333) — CLS is structural so it will not flake on slower
# fleet hardware, and the frame budget carries ~3x margin (p95 ~10ms ≤ 33ms).
# Both gates self-verify their detector's teeth: each injects a real non-transient
# shift and asserts it is flagged, so a 0.0000 can never be a dead-observer pass.
#
# Path-gated to packages/ui (and the app-side gate + matrix) so it only runs when
# the surface it covers changes. The e2e runners bundle the fixtures with esbuild
# and stub @elizaos/core, so no core build is needed; the render-parity vitest
# imports the real @elizaos/core, so the shared i18n data is generated first.
on:
pull_request:
branches: [main]
paths:
- ".github/workflows/chat-shell-gestures.yml"
- "package.json"
- "bun.lock"
- "packages/ui/**"
- "packages/shared/**"
- "packages/core/src/i18n/**"
- "packages/app/test/chat-gesture-coverage.test.ts"
- "packages/app/docs/CHAT_GESTURE_COVERAGE.md"
push:
branches: [main, develop]
paths:
- ".github/workflows/chat-shell-gestures.yml"
- "package.json"
- "bun.lock"
- "packages/ui/**"
- "packages/shared/**"
- "packages/core/src/i18n/**"
- "packages/app/test/chat-gesture-coverage.test.ts"
- "packages/app/docs/CHAT_GESTURE_COVERAGE.md"
workflow_dispatch:
# CI fan-out fix: the PR-scoped concurrency from #14069 fell back to
# github.run_id on push, so every develop push got a unique group and
# nothing ever superseded. A 37-merge wave queued thousands of push runs
# and starved the self-hosted fleet. Fall back to github.ref (all develop
# pushes share a group) and cancel superseded runs on push too. PR behavior
# is unchanged (pull_request.number still wins the key); merge_group and
# schedule events are not in the cancel set, so the merge queue and nightly
# runs always complete.
concurrency:
group: chat-shell-gestures-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }}
permissions:
contents: read
env:
CI: "true"
BUN_VERSION: "canary"
NODE_VERSION: "24.15.0"
NODE_NO_WARNINGS: "1"
jobs:
chat-shell-gestures:
name: Chat shell gesture + parity e2e
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 40
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install --ignore-scripts --no-frozen-lockfile
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
# The render-parity contract imports the real @elizaos/core, whose i18n
# module loads generated keyword data that is gitignored. Generate it so
# the vitest run resolves (mirrors the attachment-journey-videos lane).
- name: Ensure generated shared i18n data
run: node packages/app-core/scripts/ensure-shared-i18n-data.mjs
- name: Install Playwright chromium + webkit
run: bunx playwright install --with-deps chromium webkit
# Boot-free coverage gate: fails when a gesture-handler site in
# packages/ui/src lacks a CHAT_GESTURE_COVERAGE matrix row (#12188).
- name: Chat gesture coverage gate
run: bun run --cwd packages/app test -- chat-gesture-coverage
# Component-tree render parity (jsdom/vitest) — ThreadLine vs MessageContent
# render the same structure for a shared message corpus.
- name: Render-parity contract
run: bun run --cwd packages/ui test -- src/components/chat/render-parity.contract.test.tsx
# Conversation-nav interleaving fuzz (jsdom) — the pure most-recent-first
# invariants across new / select sequences (#10042). Chat-to-chat swipe
# was removed by the single-infinite-thread redesign (#13531), so the swipe
# e2e + its jank hook were retired in #13826; the remaining overlay gesture
# cost is now covered by the maximize/restore perf gate below.
- name: Conversation-nav unit + interleaving
run: bun run --cwd packages/ui test -- src/components/shell/conversation-nav.test.ts
# The iOS-style chat-sheet detent/gesture state machine (mouse + touch).
- name: Chat-sheet gesture e2e
run: bun run --cwd packages/ui test:chat-sheet-e2e
- name: Chat-sheet Safari/WebKit gesture smoke
run: bun run --cwd packages/ui test:chat-sheet-safari-e2e
# "can't scroll chat on web" repro (#chat-scroll-web): seeds a long
# transcript, opens the sheet to FULL, and asserts the transcript scroller
# resolves a BOUNDED clientHeight (so there is overflow) and that a real
# vertical finger-drag moves scrollTop natively instead of being hijacked
# into the sheet pull. RED before the fix, GREEN after.
- name: Chat-scroll web e2e
run: bun run --cwd packages/ui test:chat-scroll-web-e2e
# Transcript is vertical-only (#14328): an over-wide child + diagonal wheel
# must not pan #continuous-thread sideways; a designed inner scroller still
# scrolls in its own row and does not chain. Both engines — the axis lock is
# a *desktop trackpad* deltaX bug and WebKit's scroll semantics differ, so
# the acceptance bar requires the assertion to hold on WebKit too.
- name: Chat transcript axis-lock e2e (chromium)
run: bun run --cwd packages/ui test:chat-scroll-axis-lock-e2e
- name: Chat transcript axis-lock e2e (webkit)
run: ENGINE=webkit bun run --cwd packages/ui test:chat-scroll-axis-lock-e2e
# TopicGroup flick-collapse/expand gestures.
- name: TopicGroup gesture e2e
run: bun run --cwd packages/ui test:chatux-gesture-e2e
# Home↔launcher pager swipe, notification pull, launcher long-press
# (real CDP touch on mobile + fine-pointer desktop edge buttons).
- name: Home-screen gesture e2e
run: bun run --cwd packages/ui test:home-screen-e2e
# Seeded launcher long-loop (#12375): 500 real CDP-touch gestures over the
# composed home↔launcher surface, every §D invariant checked after each
# command, with a full walkthrough video. Pinned seed for a deterministic
# gate; fuzz locally by unsetting ELIZA_LOOP_SEED.
- name: Launcher gesture long-loop e2e
env:
ELIZA_LOOP_SEED: "12375"
run: bun run --cwd packages/ui test:launcher-loop-e2e
# Bottom-bar state-machine gestures.
- name: Bottombar gesture e2e
run: bun run --cwd packages/ui test:bottombar-e2e
# Warm-up 503 eviction: optimistic bubble survives the reconcile reload +
# retry-once delivery (records the recovery flow).
- name: Warm-up eviction e2e
run: bun run --cwd packages/ui test:warmup-eviction-e2e
# /chat ambient orange-pulse background: sample the 30s CSS animation
# across its named peaks while recording a .webm.
- name: Chat ambient pulse e2e
run: bun run --cwd packages/ui test:chat-ambient-e2e
# Perf gates: drive the overlay under thread-scroll + pull-to-maximize /
# top-pull-restore, feed real PerformanceObserver + rAF entries into the
# shared frame-budget + layout-stability detectors, hard-fail on dropped-
# frame % / p95 / CLS. Each gate first proves its CLS detector has teeth by
# injecting a real non-transient shift and asserting it is flagged (#14333),
# so a true CLS of 0.0000 cannot be a vacuous pass from a dead observer.
- name: Chat overlay perf gate
run: bun run --cwd packages/ui test:chat-perf-gate
# Scroll + maximize/restore perf gate over the REAL overlay (#9954 item 5,
# retargeted off the removed swipe path by #13826). Hard-fails on
# dropped-frame % / p95 frame time / CLS ≤ 0.1. Re-enabled in this lane by
# #14333 now that the swipe-CLS 0.80 is gone (CLS 0.0000 on develop).
- name: Chat overlay scroll/maximize perf gate
run: bun run --cwd packages/ui test:perf-gate-e2e
# #9142 Part 2: capture a dense CDP frame burst across the pill→open spring
# (animations ON) and run the single-frame-flash + two-pills-crossfade
# detectors. The harness was merged (#10356/#10374) but ran in NO workflow,
# so a transient glitch could still ship. Run under bun (resolves the
# esbuild/pixelmatch/pngjs deps directly, regardless of the node-run
# portability fallback in #10400).
- name: Frame-glitch (transient) e2e
run: bun packages/ui/src/components/shell/__e2e__/run-chat-sheet-frame-glitch-e2e.mjs
# Self-test: --canary injects a one-frame wrong state and asserts the
# detectors FIRE on it; the harness exits 0 iff they do (and non-zero if it
# has gone blind). This guards against the detectors silently degrading.
- name: Frame-glitch canary self-test
run: bun packages/ui/src/components/shell/__e2e__/run-chat-sheet-frame-glitch-e2e.mjs --canary
- name: Upload gesture videos + screenshots
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: chat-shell-gesture-artifacts
path: |
packages/ui/src/components/shell/__e2e__/output/
packages/ui/src/components/shell/__e2e__/output-webkit/
packages/ui/src/components/shell/__e2e__/output-chatux/
packages/ui/src/components/shell/__e2e__/output-home/
packages/ui/src/components/shell/__e2e__/output-launcher-loop/
packages/ui/src/components/shell/__e2e__/output-bottombar/
packages/ui/src/components/shell/__e2e__/output-warmup-eviction/
packages/ui/src/components/shell/__e2e__/output-perf-gate/
packages/ui/src/components/shell/__e2e__/output-frame-glitch/
if-no-files-found: warn
retention-days: 14
@@ -0,0 +1,81 @@
name: CI Full Matrix Proof
# Un-cancellable proof that the exhaustive develop lane keeps its full coverage
# shape (#12342). This does NOT re-run the test matrix — `test.yml` owns that on
# its own schedule. It runs the deterministic proof job: it enumerates every
# expected lane from the committed manifest, cross-checks them against
# `test.yml` (job present, not pinned pull_request-only) and against
# `run-all-tests.mjs --plan=json` (task/package floors, required core packages,
# non-empty per-script lanes). A missing lane, a lane pointed at a nonexistent
# glob, or a whole script lane collapsing to zero turns this red.
#
# Additive by design: it introduces no new required status check and does not
# touch any PR-critical workflow. The twice-daily cadence exists so the #12342
# "≥ twice daily for 7 consecutive days" observation window has a dedicated,
# never-cancelled signal to sample.
on:
schedule:
# Twice daily, offset from test.yml's 09:17 nightly so the two do not stampede.
- cron: "13 3 * * *"
- cron: "13 15 * * *"
workflow_dispatch:
# Never cancel an in-flight proof run. Push/PR traffic must not be able to
# abort the scheduled exhaustive-lane signal (#12337/#12342).
concurrency:
group: ci-full-matrix-proof-${{ github.event_name }}-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: read
env:
CI: "true"
# One pinned Bun source, matching test.yml. Floating canary hangs `bun install`
# on hosted runners and writes a lockfileVersion that breaks --frozen-lockfile
# (#11184/#9454), which would itself masquerade as a failed proof.
BUN_VERSION: "1.3.14"
NODE_VERSION: "24.15.0"
NODE_NO_WARNINGS: "1"
NODE_OPTIONS: "--max-old-space-size=8192"
jobs:
matrix-proof:
name: Exhaustive lane matrix proof
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install --frozen-lockfile --ignore-scripts
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Validate GitHub-native turbo cache contract
run: node packages/scripts/ci-turbo-cache-contract.mjs
- name: Capture exhaustive test plan
run: node packages/scripts/run-all-tests.mjs --plan=json > /tmp/full-matrix-plan.json
- name: Prove exhaustive lane coverage
run: node packages/scripts/ci-full-matrix-proof.mjs --plan-file /tmp/full-matrix-plan.json
- name: Upload captured plan
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: full-matrix-plan
path: /tmp/full-matrix-plan.json
if-no-files-found: warn
+245
View File
@@ -0,0 +1,245 @@
name: ci
# Cancel previous runs for the same PR/branch
concurrency:
group: ci-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
on:
push:
branches: [main]
pull_request:
branches: [main]
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
# Test job
test:
# Skip duplicate runs: run on push to main, or on pull_request events only
if: github.event_name == 'pull_request' || (github.event_name == 'push' && github.ref_name == 'main')
runs-on: ubuntu-latest
# 45 not 35: typecheck runs at --concurrency=4 (see below, #15140), so a
# near-total affected cone (e.g. a develop→main promote PR) needs headroom
# on top of install + build + the 15-minute test step.
timeout-minutes: 45
env:
# Baseline CI is secret-free. Live/provider-key E2E runs in dedicated
# opt-in or post-merge workflows, not in the PR test job.
PGLITE_WASM_MODE: node
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
# Full history so `turbo run typecheck --affected` can resolve the PR
# merge base (#12341); a shallow clone degrades to typechecking all.
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24.15.0"
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: "1.3.14" # pinned: floating canary writes lockfileVersion 2 and breaks --frozen-lockfile (#11184/#9454); packageManager bun@1.4.0-canary.1 is unresolvable (GH+npm 404)
install-command: bun install
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Create test env file
run: |
echo "TEST_DATABASE_CLIENT=pglite" > packages/core/.env.test
echo "NODE_ENV=test" >> packages/core/.env.test
- name: Build packages
run: bun run build:core
# PR lane: typecheck only the merge base's dependency cone; on push to
# main typecheck the whole workspace (#12341). --concurrency=4 not 8:
# a develop→main promote PR's affected cone is near-total, and eight
# concurrent tsgo processes exhaust a 16 GB hosted runner (#15140);
# mirrors develop-pr.yml so the PR lanes stay in agreement.
- name: Run typecheck (affected — PR)
if: github.event_name == 'pull_request'
run: NODE_OPTIONS='--max-old-space-size=8192' node packages/scripts/run-turbo.mjs run typecheck --concurrency=4 --affected
env:
TURBO_SCM_BASE: ${{ github.event.pull_request.base.sha }}
- name: Run typecheck (full — push)
if: github.event_name != 'pull_request'
run: bun run typecheck
- name: Run tests
timeout-minutes: 15
env:
NODE_OPTIONS: "--max-old-space-size=2048"
run: bun run test:core
- name: Run plugin tests
timeout-minutes: 15
env:
NODE_OPTIONS: "--max-old-space-size=2048"
run: bun run test:plugins
- name: Run interop (TypeScript) tests
timeout-minutes: 10
run: |
if [ -d packages/interop ]; then
cd packages/interop && bun run test
else
echo "packages/interop is not present; skipping interop tests"
fi
# Lint and format job
lint-and-format:
# Skip duplicate runs: run on push to main, or on pull_request events only
if: github.event_name == 'pull_request' || (github.event_name == 'push' && github.ref_name == 'main')
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
# `audit:test-realness` compares touched tests against origin/develop.
# Keep the merge base available so the diff-scoped ratchet runs in CI.
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24.15.0"
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: "1.3.14" # pinned: floating canary writes lockfileVersion 2 and breaks --frozen-lockfile (#11184/#9454); packageManager bun@1.4.0-canary.1 is unresolvable (GH+npm 404)
install-command: bun install
install-native-deps: "false"
install-protoc: "false"
setup-python: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Prompt secret scan
run: cd packages/prompts && bun run check:secrets
- name: Type safety ratchet self-test
run: bun run audit:type-safety-ratchet:self-test
- name: Type safety ratchet
run: bun run audit:type-safety-ratchet
- name: Check format
run: bun run format:check
- name: Run lint
run: bun run lint
# UI render-time determinism gate. Fails on NEW Date.now()/Math.random()/
# crypto.randomUUID()/locale-defaulted toLocale* in a component/hook render
# path (the source of flaky screenshots + meaningless snapshots). The
# existing backlog is tracked in packages/scripts/ui-determinism-baseline.json;
# only regressions beyond it fail. Self-test asserts the AST classifier.
- name: UI determinism self-test
run: bun run audit:ui-determinism:self-test
- name: UI determinism gate
run: bun run audit:ui-determinism
# Anti-larp test gate (#10718). Fails on focused tests (.only / fit /
# fdescribe — a single .only silently drops every sibling test in the file)
# and on orphaned hardcoded skips (it.skip("name", fn) with no reason,
# tracking issue, deny-list ref, or Playwright skip-annotation). Legitimate
# conditional/env-gated skips (cond ? describe : describe.skip,
# test.skip(cond, reason)) are allowed. Self-test asserts the classifier.
- name: Anti-larp test gate self-test
run: bun run audit:focused-tests:self-test
- name: Anti-larp test gate
run: bun run audit:focused-tests
- name: Test realness ratchet
run: bun run audit:test-realness
# Per-plugin mock-LLM e2e coverage. The seeded allowlist carries
# historical debt; new unsuppressed gaps fail the main gate (#13620).
- name: Per-plugin e2e coverage
run: node packages/scripts/lint-lane-coverage.mjs
# Build job
build:
# Skip duplicate runs: run on push to main, or on pull_request events only
if: github.event_name == 'pull_request' || (github.event_name == 'push' && github.ref_name == 'main')
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24.15.0"
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: "1.3.14" # pinned: floating canary writes lockfileVersion 2 and breaks --frozen-lockfile (#11184/#9454); packageManager bun@1.4.0-canary.1 is unresolvable (GH+npm 404)
install-command: bun install
install-native-deps: "false"
install-protoc: "false"
setup-python: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Build packages
run: bun run build:core
# Dev startup + HMR job: prove `bun run dev` boots to a usable state quickly
# and that edits at every package dependency depth propagate to the running
# client over HMR. Guards against startup-time regressions and broken
# src/-resolution or workspace-source watching.
dev-startup:
if: github.event_name == 'pull_request' || (github.event_name == 'push' && github.ref_name == 'main')
runs-on: ubuntu-latest
timeout-minutes: 25
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24.15.0"
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: "1.3.14" # pinned: floating canary writes lockfileVersion 2 and breaks --frozen-lockfile (#11184/#9454); packageManager bun@1.4.0-canary.1 is unresolvable (GH+npm 404)
install-command: bun install
install-native-deps: "false"
install-protoc: "false"
setup-python: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Build packages
run: bun run build
- name: Remove source-adjacent build output
run: bun run clean:stale-js
- name: Install Playwright Chromium
run: bunx playwright install --with-deps chromium
# The product gate is 60s (the script default enforced on dev machines).
# Shared CI runners are ~2-4 cores vs a dev workstation, so the CI ceiling
# is relaxed to 90s — still catches a startup-time doubling, without
# flaky false-failures. Lower this if runner class improves.
- name: Dev startup budget (boots + ready under budget)
env:
ELIZA_DEV_STARTUP_BUDGET_MS: "90000"
run: bun run test:dev-startup
- name: HMR propagation across dependency levels
run: bun run test:hmr
+96
View File
@@ -0,0 +1,96 @@
name: Claude Code Review
# Cancel previous runs for the same PR/branch
concurrency:
group: claude-code-review-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
on:
pull_request:
types: [opened, ready_for_review, synchronize, reopened]
# Allow manual triggering for when you specifically want a review
workflow_dispatch:
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
claude-review:
# Skip review if PR title contains [skip-review] or is a draft
if: |
!contains(github.event.pull_request.title, '[skip-review]') &&
github.event.pull_request.draft != true
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 15
permissions:
contents: read
pull-requests: write
issues: read
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 1
- name: Run Claude Code Review
id: claude-review
# Don't fail the workflow when the Anthropic API is unavailable
# (low credit balance, rate limit, transient outage). The bot review
# is best-effort; a missing review must not block CI.
continue-on-error: true
uses: anthropics/claude-code-action@ba0aafd4308cbba7165f9f2cdb0cfbed5a3c99ce
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
track_progress: true
# Prompt for automated review (no @claude mention needed)
prompt: |
REPO: ${{ github.repository }}
PR NUMBER: ${{ github.event.pull_request.number }}
Review this PR. Be concise and actionable.
**📝 IMPORTANT: Check and consider all previous review comments in this PR before making new suggestions. Don't repeat issues that have already been addressed or are being discussed.**
**🚨 CRITICAL CHECKS:**
- Security: hardcoded keys, SQL injection, XSS
- No tests = REJECT (untested code is broken)
- Wrong tools: npm/pnpm/yarn/jest = REJECT (use vitest)
- Breaking changes without migration = REJECT
**✅ REQUIRED:**
- TypeScript types (no 'any')
- Tests with vitest (via bun run test)
- Use @elizaos/core imports (canonical sources live under packages/core)
- Functional code (no classes)
- Error handling
**📋 VERIFY:**
- All new code has tests
- Use bun for scripts, vitest for testing
- No circular dependencies
- Follows existing patterns
- Docs updated if needed
**🎯 OUTPUT FORMAT:**
Provide detailed feedback using inline comments for specific issues.
```
❌ CRITICAL: [issue] → Fix: [specific action]
⚠️ IMPORTANT: [issue] → Fix: [specific action]
💡 SUGGESTION: [improvement] → Consider: [action]
```
Skip explanations. List issues with fixes.
# Allow cursor bot to trigger reviews
allowed_bots: "cursor"
# Claude CLI arguments for model and allowed tools
claude_args: |
--model claude-opus-4-7
--allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment *),Bash(gh pr diff *),Bash(gh pr view *),Bash(gh api *)"
@@ -0,0 +1,69 @@
name: Security Review
on:
pull_request:
types: [opened, synchronize, reopened]
# Focus on code changes that could introduce vulnerabilities
paths:
- "**/*.ts"
- "**/*.tsx"
- "**/*.js"
- "**/*.jsx"
- "**/*.json"
- "**/*.yml"
- "**/*.yaml"
- "**/Dockerfile"
- "**/.env.example"
# Allow manual security scans
workflow_dispatch:
# Cancel previous runs for the same PR
concurrency:
group: security-review-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
security:
# Skip for draft PRs, [skip-security] in title, or bot-triggered
if: |
github.event.pull_request.draft != true &&
!contains(github.event.pull_request.title, '[skip-security]') &&
github.actor != 'cursor[bot]' &&
github.actor != 'dependabot[bot]'
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 15
permissions:
contents: read
pull-requests: write
security-events: write
steps:
- name: Checkout repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
ref: ${{ github.event.pull_request.head.sha || github.sha }}
fetch-depth: 2
- name: Run Claude Security Review
if: ${{ env.ANTHROPIC_API_KEY != '' }}
# Pinned to SHA for supply chain security.
uses: anthropics/claude-code-security-review@0c6a49f1fa56a1d472575da86a94dbc1edb78eda
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
with:
claude-api-key: ${{ env.ANTHROPIC_API_KEY }}
comment-pr: true
upload-results: true
claudecode-timeout: 30
# Exclude non-critical directories
exclude-directories: "node_modules,dist,coverage,.turbo,.next,docs"
# Use Opus 4.5 for deeper security analysis
claude-model: claude-opus-4-7
+55
View File
@@ -0,0 +1,55 @@
name: Claude Code
on:
issue_comment:
types: [created]
pull_request_review_comment:
types: [created]
issues:
types: [opened, assigned]
pull_request_review:
types: [submitted]
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
claude:
if: |
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
(github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 15
permissions:
contents: read
pull-requests: write
issues: write
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 1
- name: Run Claude Code
id: claude
# Don't fail the workflow when the Anthropic API is unavailable
# (low credit balance, rate limit, transient outage). @claude is a
# best-effort assistant; an unavailable run must not turn the CI red.
continue-on-error: true
uses: anthropics/claude-code-action@ba0aafd4308cbba7165f9f2cdb0cfbed5a3c99ce
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
# Allow cursor bot to trigger reviews
allowed_bots: "cursor"
# Claude CLI arguments for model, allowed tools, and system prompt
claude_args: |
--model claude-opus-4-7
--allowedTools "Bash(*),Bash(gh *)"
--system-prompt "Be concise. Focus on actionable solutions. Use bun only (no npm/pnpm/yarn). Use vitest for testing (no jest). Check logs: gh run view <run-id> --log. Follow CLAUDE.md standards. IMPORTANT: When reviewing PRs, always check and consider all previous comments in the PR thread. Don't repeat suggestions that have already been made or addressed."
@@ -0,0 +1,92 @@
name: Cloud Aesthetic Audit
# Walk EVERY registered Eliza Cloud route (dashboard, payment, auth, admin, …) at
# desktop + mobile against the deterministic keyless stub stack with the
# test-auth shell baked into the renderer, capture rest + hover screenshots, run
# the blank/one-color analyzer + brand-color (no-blue / orange-hover) checks, and
# write per-view verdicts + a contact sheet + report.json.
#
# This is the cloud-surface twin of `app-aesthetic-audit.yml`. Until it existed
# `audit:cloud` ran in NO CI lane and was a pure reporter — a `broken`/`needs-work`
# cloud page failed nothing, and a turbo-cached renderer built WITHOUT the
# test-auth shell made the whole suite skip green with ZERO pages walked (#13624,
# the same class #9304 fixed for audit:app).
#
# Under `ELIZA_AUDIT_CLOUD_STRICT=1` (+ CI auto-on) the audit is a GATE: an
# undebted `broken` view fails; a missing auth shell or an empty walk is a HARD
# FAILURE, not a skip. A CLEAN renderer rebuild with VITE_PLAYWRIGHT_TEST_AUTH
# baked in guarantees the shell is present regardless of the turbo cache.
on:
pull_request:
branches: [main]
paths:
- "packages/app/**"
- "packages/ui/**"
- "packages/shared/**"
- "packages/cloud/**"
- ".github/workflows/cloud-aesthetic-audit.yml"
workflow_dispatch:
concurrency:
group: cloud-aesthetic-audit-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
BUN_VERSION: "canary"
NODE_VERSION: "24.15.0"
jobs:
cloud-aesthetic-audit:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 45
env:
# Keyless: the ui-smoke stub is forced when CI=true (shouldForceStubStack).
ELIZA_UI_SMOKE_FORCE_STUB: "1"
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Install Playwright Chromium
run: bunx playwright install --with-deps chromium
# Force a clean renderer build with the test-auth shell baked in. A stale
# dist (built without VITE_PLAYWRIGHT_TEST_AUTH) is the exact #13624 hole —
# remove it so the audit cannot walk a shell-less bundle.
- name: Clean renderer dist
run: rm -rf packages/app/dist
- name: Run cloud aesthetic audit (strict)
# audit:cloud sets VITE_PLAYWRIGHT_TEST_AUTH=true, so the build:web it
# triggers bakes the shell. ELIZA_AUDIT_CLOUD_STRICT=1 turns the reporter
# into a gate: undebted `broken` fails, missing shell / empty walk fails.
env:
ELIZA_AUDIT_CLOUD_STRICT: "1"
run: bun run --cwd packages/app audit:cloud
- name: Upload cloud aesthetic-audit artifacts
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: cloud-aesthetic-audit-output
path: packages/app/aesthetic-audit-output-cloud
retention-days: 14
if-no-files-found: ignore
File diff suppressed because it is too large Load Diff
+431
View File
@@ -0,0 +1,431 @@
name: Cloud Deploy Backend
# Source: cloud/.github/workflows/deploy-backend.yml in the original elizaOS/cloud repo.
#
# This is a manual-only legacy operations workflow. Canonical Cloudflare
# Worker/Pages releases and their database migrations are owned by
# cloud-cf-deploy.yml; dispatch this workflow only for an explicit standalone
# migration or legacy VPS deployment.
on:
workflow_dispatch:
inputs:
environment:
description: 'Environment to deploy'
required: true
default: 'staging'
type: choice
options:
- staging
- production
deploy_legacy_vps:
description: 'Run the legacy VPS deploy after migrations'
required: false
default: false
type: boolean
concurrency:
group: cloud-deploy-backend-${{ github.ref }}
cancel-in-progress: false
env:
BUN_VERSION: "canary"
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
determine-env:
name: Determine Environment
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
outputs:
environment: ${{ steps.env.outputs.environment }}
branch: ${{ steps.env.outputs.branch }}
steps:
- name: Set environment
id: env
run: |
echo "environment=${{ github.event.inputs.environment }}" >> $GITHUB_OUTPUT
echo "branch=${{ github.event.inputs.environment == 'production' && 'main' || 'develop' }}" >> $GITHUB_OUTPUT
migrate-db:
name: Run Database Migrations
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
needs: determine-env
# Job-level concurrency groups are repo-wide: this serializes an explicit
# legacy migration against the canonical migrate-db gate in
# cloud-cf-deploy.yml (#11208).
concurrency:
group: cloud-db-migrate-${{ needs.determine-env.outputs.environment }}
cancel-in-progress: false
environment: ${{ needs.determine-env.outputs.environment }}
timeout-minutes: 10
env:
MIGRATION_DATABASE_URL: ${{ secrets.DATABASE_URL || secrets.RAILWAY_DATABASE_URL || secrets.NEON_DATABASE_URL }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install dependencies
run: |
for attempt in 1 2 3; do
if [ "$attempt" -eq 3 ]; then
bun install --no-save --ignore-scripts --verbose && exit 0
else
bun install --no-save --ignore-scripts && exit 0
fi
echo "bun install attempt $attempt failed; retrying in 10s..."
sleep 10
done
exit 1
- name: Fail fast when database secret is missing
if: ${{ env.MIGRATION_DATABASE_URL == '' }}
run: |
# Previously this step quietly skipped migrations whenever the secret
# was unset, so 46+ migrations silently accumulated against prod from
# 2026-04-22 onward (sukimyfun Stripe webhook 500 RCA, 2026-05-20).
# Fail loudly instead — a deploy that ships code expecting a newer
# schema must not succeed if the DB can't be migrated to match.
echo "::error::DATABASE_URL / RAILWAY_DATABASE_URL / NEON_DATABASE_URL secret is missing on the '${{ needs.determine-env.outputs.environment }}' environment. The schema cannot be migrated, so the worker would deploy against a stale DB."
echo "Add the secret in: Settings → Environments → ${{ needs.determine-env.outputs.environment }} → Environment secrets."
exit 1
- name: Run migrations
env:
DATABASE_URL: ${{ env.MIGRATION_DATABASE_URL }}
run: |
# The migrate entry imports @elizaos/core, whose i18n barrel pulls in the
# gitignored generated keyword data. Source-mode (eliza-source) migrate
# never builds core, so generate the file first (mirrors core's prebuild).
node packages/shared/scripts/generate-keywords.mjs
bun run db:cloud:migrate
- name: Notify Discord (Success)
if: success()
uses: sarisia/actions-status-discord@eb045afee445dc055c18d3d90bd0f244fd062708
with:
webhook: ${{ secrets.DISCORD_WEBHOOK }}
ack_no_webhook: true
title: "🗄️ Database Migrated"
nodetail: true
description: |
Environment: ${{ needs.determine-env.outputs.environment }}
Branch: ${{ needs.determine-env.outputs.branch }}
Commit: ${{ github.sha }}
color: 0x00ff00
- name: Notify Discord (Failure)
if: failure()
uses: sarisia/actions-status-discord@eb045afee445dc055c18d3d90bd0f244fd062708
with:
webhook: ${{ secrets.DISCORD_WEBHOOK }}
ack_no_webhook: true
title: "❌ Database Migration Failed"
nodetail: true
description: |
Environment: ${{ needs.determine-env.outputs.environment }}
Branch: ${{ needs.determine-env.outputs.branch }}
Commit: ${{ github.sha }}
color: 0xff0000
deploy:
name: Deploy to agent VPS
if: ${{ github.event_name == 'workflow_dispatch' && inputs.deploy_legacy_vps }}
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
needs: [determine-env, migrate-db]
environment: ${{ needs.determine-env.outputs.environment }}
timeout-minutes: 45
env:
DEPLOY_VPS_HOST: ${{ secrets.AGENT_VPS_HOST || secrets.ELIZA_VPS_HOST }}
DEPLOY_VPS_SSH_KEY: ${{ secrets.AGENT_VPS_SSH_KEY || secrets.ELIZA_VPS_SSH_KEY }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install dependencies
run: |
for attempt in 1 2 3; do
if [ "$attempt" -eq 3 ]; then
bun install --no-save --ignore-scripts --verbose && exit 0
else
bun install --no-save --ignore-scripts && exit 0
fi
echo "bun install attempt $attempt failed; retrying in 10s..."
sleep 10
done
exit 1
- name: Build backend artifact
run: |
bun run --cwd packages/cloud/api codegen
bun run --cwd packages/cloud/api lint
- name: Prepare SSH access
run: |
if [ -z "$DEPLOY_VPS_HOST" ] || [ -z "$DEPLOY_VPS_SSH_KEY" ]; then
echo "::error::Missing VPS deploy secrets. Configure AGENT_VPS_HOST/AGENT_VPS_SSH_KEY or ELIZA_VPS_HOST/ELIZA_VPS_SSH_KEY for the GitHub environment."
exit 1
fi
install -d -m 700 "$HOME/.ssh"
printf '%s\n' "$DEPLOY_VPS_SSH_KEY" > "$HOME/.ssh/id_ed25519"
chmod 600 "$HOME/.ssh/id_ed25519"
ssh-keyscan -H "$DEPLOY_VPS_HOST" >> "$HOME/.ssh/known_hosts"
- name: Prepare VPS checkout
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2
with:
host: ${{ env.DEPLOY_VPS_HOST }}
username: deploy
key: ${{ env.DEPLOY_VPS_SSH_KEY }}
script_stop: true
command_timeout: 30m
script: |
set -e
echo "=== Deploying eliza-cloud backend ==="
sudo chown -R deploy:deploy /opt/eliza-cloud
cd /opt/eliza-cloud
# Use env-based config to avoid writing ~/.gitconfig (lock file failures)
export GIT_CONFIG_COUNT=1
export GIT_CONFIG_KEY_0=safe.directory
export GIT_CONFIG_VALUE_0=/opt/eliza-cloud
# --- Disk cleanup (prevents "Out of diskspace" failures) ---
echo "Disk before cleanup:"
df -h / | tail -1
# Remove stale index.lock FIRST (previous failed git op leaves this)
rm -f .git/index.lock
# Remove previous build output and local build caches.
# Keep node_modules in place so bun install can reuse the existing
# tree instead of redownloading the full dependency graph onto a
# nearly-full VPS.
rm -rf .next-build .next .turbo node_modules/.cache
# Drop all accumulated deploy stashes
git stash clear 2>/dev/null || true
# Prune loose git objects
git prune 2>/dev/null || true
# Trim Bun/Turbo caches aggressively
export BUN_INSTALL="$HOME/.bun"
export PATH="$BUN_INSTALL/bin:$PATH"
rm -rf "$HOME/.bun/install/cache" "$HOME/.cache/bun" "$HOME/.cache/turbo"
if command -v bun >/dev/null 2>&1; then
bun pm cache rm 2>/dev/null || true
fi
# Free host-level disk before touching the checkout. The root
# filesystem fills up outside /opt/eliza-cloud too, especially with
# old Docker layers, package caches, and journals.
sudo apt-get clean 2>/dev/null || true
sudo journalctl --vacuum-size=500M 2>/dev/null || true
sudo docker system prune -af 2>/dev/null || true
sudo rm -rf /root/.bun/install/cache /root/.cache/bun /root/.cache/turbo 2>/dev/null || true
# Discard local checkout drift before fetch. Cleaning untracked
# files first frees space for git to rewrite tracked files.
# Preserve the server's environment file and keep dependency trees
# if there is enough free disk to reuse them.
echo "Discarding local checkout drift..."
git clean -fdx \
-e .env \
-e .env.local \
-e .env.* \
-e node_modules \
-e services/gateway-discord/node_modules \
-e services/gateway-webhook/node_modules \
-e packages/ui/node_modules \
-e services/agent-server/node_modules 2>/dev/null || true
rm -f .git/index.lock
git checkout -- . 2>/dev/null || true
rm -f .git/index.lock
# The systemd unit on the VPS expects a repo-local env file, but the
# exact filename can vary across older hosts. Preserve all root env
# files and recreate the common aliases from whichever one survived.
ENV_SOURCE=""
for candidate in .env.local .env.production .env.staging .env; do
if [ -f "$candidate" ]; then
ENV_SOURCE="$candidate"
break
fi
done
if [ -n "$ENV_SOURCE" ]; then
for alias in .env .env.local .env.production .env.staging; do
if [ ! -e "$alias" ]; then
ln -s "$ENV_SOURCE" "$alias"
fi
done
fi
echo "Env files available on VPS checkout:"
ls -la .env* 2>/dev/null || true
AVAILABLE_KB="$(df --output=avail -k / | tail -1 | tr -d ' ')"
if [ "${AVAILABLE_KB:-0}" -lt 5000000 ]; then
echo "Still under 5GB free; removing workspace node_modules as a fallback..."
rm -rf \
node_modules \
services/gateway-discord/node_modules \
services/gateway-webhook/node_modules \
packages/ui/node_modules \
services/agent-server/node_modules
fi
echo "Disk after cleanup:"
df -h / | tail -1
# --- End disk cleanup ---
# Fetch and checkout
BRANCH="${{ needs.determine-env.outputs.branch }}"
git fetch origin "$BRANCH"
git checkout -B "$BRANCH" "origin/$BRANCH"
# Install runtime dependencies and regenerate lightweight assets.
if ! command -v bun >/dev/null 2>&1; then
curl -fsSL https://bun.sh/install | bash
export PATH="$BUN_INSTALL/bin:$PATH"
fi
for attempt in 1 2 3; do
if bun install --no-save --ignore-scripts; then
break
fi
if [ "$attempt" -eq 3 ]; then
exit 1
fi
echo "bun install attempt $attempt failed; retrying in 10s..."
sleep 10
done
bun run generate:llms
# The build now runs on GitHub Actions to avoid ENOSPC during
# `next build` on the VPS. Clear any stale output and leave an empty
# destination for the streamed artifact.
rm -rf "$HOME/.bun/install/cache" "$HOME/.cache/bun"
rm -rf node_modules/.cache
rm -rf .next-build
mkdir -p .next-build
- name: Upload prebuilt Next output
run: |
# Only ship the runtime output. The webpack cache dominates .next-build
# size locally but is not needed by `next start` on the VPS.
tar -C .next-build -czf - \
BUILD_ID \
app-path-routes-manifest.json \
build-manifest.json \
export-marker.json \
images-manifest.json \
next-minimal-server.js.nft.json \
next-server.js.nft.json \
package.json \
prerender-manifest.json \
react-loadable-manifest.json \
required-server-files.js \
required-server-files.json \
routes-manifest.json \
server \
static | ssh -i "$HOME/.ssh/id_ed25519" \
deploy@"$DEPLOY_VPS_HOST" \
'set -e; cd /opt/eliza-cloud; rm -rf .next .next-build/*; tar -xzf - -C .next-build; ln -sfn .next-build .next'
- name: Restart services
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2
with:
host: ${{ env.DEPLOY_VPS_HOST }}
username: deploy
key: ${{ env.DEPLOY_VPS_SSH_KEY }}
script_stop: true
script: |
set -e
cd /opt/eliza-cloud
rm -rf .next
ln -sfn .next-build .next
if ! sudo systemctl restart eliza-cloud; then
sudo systemctl cat eliza-cloud || true
sudo systemctl show eliza-cloud \
--property=FragmentPath,EnvironmentFiles,ExecStart,WorkingDirectory || true
ls -la /opt/eliza-cloud/.env* 2>/dev/null || true
sudo systemctl status eliza-cloud --no-pager || true
sudo journalctl -u eliza-cloud -n 100 --no-pager || true
exit 1
fi
sudo systemctl restart agent-provisioning-worker
echo "=== Deploy complete ==="
- name: Health Check
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2
with:
host: ${{ env.DEPLOY_VPS_HOST }}
username: deploy
key: ${{ env.DEPLOY_VPS_SSH_KEY }}
script_stop: true
script: |
echo "Checking eliza-cloud health..."
for attempt in $(seq 1 18); do
for url in http://localhost:3000/api/health http://localhost:3334/api/health; do
if curl -sf "$url" > /dev/null; then
echo "Health check passed on attempt $attempt via $url."
exit 0
fi
done
echo "Health check attempt $attempt/18 failed; retrying in 5s..."
sleep 5
done
echo "eliza-cloud failed health checks after 90s. Collecting diagnostics..."
sudo ss -ltnp | grep -E ':(3000|3334)\\s' || true
sudo systemctl status eliza-cloud --no-pager || true
sudo journalctl -u eliza-cloud -n 100 --no-pager || true
exit 1
- name: Notify Discord
if: success()
uses: sarisia/actions-status-discord@eb045afee445dc055c18d3d90bd0f244fd062708
with:
webhook: ${{ secrets.DISCORD_WEBHOOK }}
ack_no_webhook: true
title: "🚀 Backend Deployed"
description: |
Environment: ${{ needs.determine-env.outputs.environment }}
Branch: ${{ needs.determine-env.outputs.branch }}
Commit: ${{ github.sha }}
color: 0x00ff00
- name: Notify Discord (Failure)
if: failure()
uses: sarisia/actions-status-discord@eb045afee445dc055c18d3d90bd0f244fd062708
with:
webhook: ${{ secrets.DISCORD_WEBHOOK }}
ack_no_webhook: true
title: "❌ Backend Deploy Failed"
description: |
Environment: ${{ needs.determine-env.outputs.environment }}
Branch: ${{ needs.determine-env.outputs.branch }}
Commit: ${{ github.sha }}
color: 0xff0000
+89
View File
@@ -0,0 +1,89 @@
name: cloud-e2e
# Mock-stack E2E for cloud-api + the cloud web app (packages/app).
# Runs `bun run cloud:e2e`, which boots in-process PGlite + ioredis-mock + Hetzner mock +
# control-plane mock and drives the real cloud-api / cloud web dev servers via Playwright.
on:
pull_request:
branches: [main]
paths:
- "packages/cloud/api/**"
- "packages/cloud/shared/**"
- "packages/cloud/services/**"
- "packages/cloud/sdk/**"
- "packages/test/cloud-mocks/**"
- "packages/test/cloud-e2e/**"
# The showcase loop (#9300) deploys + validates these example apps, so an
# edit to either app must trigger the showcase e2e + brand-lint lane.
- "packages/examples/cloud/**"
- ".github/workflows/cloud-e2e.yml"
workflow_dispatch:
concurrency:
group: cloud-e2e-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
e2e:
name: Mock-stack E2E
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 30
env:
NODE_ENV: test
NODE_OPTIONS: --max-old-space-size=4096
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
with:
fetch-depth: 1
- uses: ./.github/actions/cloud-setup-test-env
- name: Install Playwright browsers (chromium only)
run: bunx playwright install --with-deps chromium
- name: Build cloud-shared (drizzle artifacts)
# cloud-e2e fixture re-runs migrations against PGlite so build is best-effort.
run: bun run --cwd packages/cloud/shared build || true
- name: Run cloud E2E
run: bun run cloud:e2e
env:
CI: "true"
CLOUD_E2E: "1"
# mocks read these
MOCK_REDIS: "1"
MOCK_HETZNER_LATENCY: "0"
MOCK_HETZNER_ACTION_MS: "30"
CONTROL_PLANE_TICK_MS: "50"
# placeholder values so client code doesn't crash on lookup; no real creds
DATABASE_URL: "pglite://./.eliza-ci/.pgdata"
HCLOUD_TOKEN: "test-token"
CONTAINER_CONTROL_PLANE_TOKEN: "test-token"
CRON_SECRET: "test-cron-secret"
- name: Upload Playwright artifacts
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: cloud-e2e-playwright-${{ github.run_attempt }}
path: |
packages/test/cloud-e2e/playwright-report
packages/test/cloud-e2e/test-results
retention-days: 7
if-no-files-found: ignore
- name: Upload process logs (boot diagnostics)
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: cloud-e2e-process-logs-${{ github.run_attempt }}
path: |
packages/test/cloud-e2e/.logs
retention-days: 7
include-hidden-files: true
if-no-files-found: ignore
@@ -0,0 +1,99 @@
name: Cloud Gateway Discord
# CI for the gateway-discord service. The service is deployed to Railway
# (see packages/cloud/services/gateway-discord/railway.toml); Railway auto-
# deploys on push to the watched branch, so this workflow only runs tests
# and a build sanity check.
#
# AWS EKS / Terraform / Helm deploy jobs were removed as part of the AWS
# retirement (see packages/cloud/infra/cloud/AWS_RETIREMENT.md).
on:
push:
branches: [main, develop]
paths:
- "packages/cloud/services/gateway-discord/**"
- "packages/cloud/shared/src/lib/services/gateway-discord/**"
- "packages/cloud/shared/src/lib/auth/jwks.ts"
- "packages/cloud/shared/src/lib/auth/jwt-internal.ts"
- "packages/cloud/shared/src/lib/auth/internal-api.ts"
- "packages/cloud/shared/scripts/messaging-gateway-preflight.mjs"
- "packages/cloud/api/v1/discord/**"
- "packages/cloud/api/internal/discord/**"
- "packages/cloud/api/internal/auth/**"
- "packages/cloud/api/.well-known/jwks.json/**"
- "packages/cloud/shared/src/db/**/discord-connections.ts"
- ".github/workflows/cloud-gateway-discord.yml"
pull_request:
branches: [main]
paths:
- "packages/cloud/services/gateway-discord/**"
- "packages/cloud/shared/src/lib/services/gateway-discord/**"
- "packages/cloud/shared/src/lib/auth/jwks.ts"
- "packages/cloud/shared/src/lib/auth/jwt-internal.ts"
- "packages/cloud/shared/src/lib/auth/internal-api.ts"
- "packages/cloud/shared/scripts/messaging-gateway-preflight.mjs"
- "packages/cloud/api/v1/discord/**"
- "packages/cloud/api/internal/discord/**"
- "packages/cloud/api/internal/auth/**"
- "packages/cloud/api/.well-known/jwks.json/**"
- "packages/cloud/shared/src/db/**/discord-connections.ts"
- ".github/workflows/cloud-gateway-discord.yml"
concurrency:
group: cloud-gateway-discord-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
defaults:
run:
working-directory: packages/cloud/shared
env:
BUN_VERSION: "canary"
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
test:
name: Test
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install root dependencies
run: bun install --no-save && git diff --exit-code -- bun.lock
- name: Run messaging gateway preflight
run: bun run preflight:messaging-gateways
- name: Install service dependencies
working-directory: packages/cloud/services/gateway-discord
run: bun install --no-save && git diff --exit-code -- bun.lock
- name: Run service tests
working-directory: packages/cloud/services/gateway-discord
run: bun test tests/ --timeout 60000
env:
SKIP_SERVER_CHECK: "true"
- name: Run lib tests
run: bun test src/lib/services/gateway-discord/__tests__ --timeout 60000
env:
SKIP_SERVER_CHECK: "true"
- name: Run Discord API unit tests
run: |
if [ -f packages/tests/unit/discord-connections-api.test.ts ]; then
bun test packages/tests/unit/discord-connections-api.test.ts --timeout 60000
else
echo "packages/tests/unit/discord-connections-api.test.ts is not present; skipping legacy Discord API unit tests"
fi
env:
SKIP_SERVER_CHECK: "true"
@@ -0,0 +1,64 @@
name: Cloud Gateway Webhook
# Source: cloud/.github/workflows/gateway-webhook.yml in the original elizaOS/cloud repo.
on:
push:
branches: [main, develop]
paths:
- "packages/cloud/services/gateway-webhook/**"
- "packages/cloud/shared/src/lib/services/**"
- "packages/cloud/shared/src/lib/onboarding/**"
- "packages/cloud/shared/src/lib/auth/**"
- "packages/cloud/shared/src/db/**"
- "packages/cloud/shared/scripts/messaging-gateway-preflight.mjs"
- ".github/workflows/cloud-gateway-webhook.yml"
pull_request:
branches: [main]
paths:
- "packages/cloud/services/gateway-webhook/**"
- "packages/cloud/shared/src/lib/services/**"
- "packages/cloud/shared/src/lib/onboarding/**"
- "packages/cloud/shared/src/lib/auth/**"
- "packages/cloud/shared/src/db/**"
- "packages/cloud/shared/scripts/messaging-gateway-preflight.mjs"
- ".github/workflows/cloud-gateway-webhook.yml"
concurrency:
group: cloud-gateway-webhook-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
defaults:
run:
working-directory: packages/cloud/shared
env:
BUN_VERSION: "canary"
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
test:
name: Test
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install dependencies
run: bun install --no-save && git diff --exit-code -- bun.lock
- name: Run messaging gateway preflight
run: bun run preflight:messaging-gateways
- name: Run unit tests (isolated — mock.module is process-global in bun)
working-directory: packages/cloud/services/gateway-webhook
run: |
bun test ./__tests__/build-forward-body.test.ts --timeout 60000
bun test ./__tests__/internal-auth.test.ts --timeout 60000
+142
View File
@@ -0,0 +1,142 @@
name: Cloud Tests
# Runs cloud's lint/typecheck/unit/integration/playwright suite when cloud-* packages or tests change.
# Source: cloud/.github/workflows/tests.yml in the original elizaOS/cloud repo.
on:
workflow_dispatch:
pull_request:
branches: [develop, main]
paths:
- ".github/actions/cloud-setup-test-env/**"
- ".github/workflows/cloud-tests.yml"
- "packages/cloud/api/**"
- "packages/cloud/shared/**"
- "packages/cloud/sdk/**"
- "packages/cloud/services/**"
- "packages/cloud/routing/**"
- "packages/cloud/infra/**"
- "packages/scripts/cloud/**"
- "packages/scripts/test-cloud-run.mjs"
- "packages/scripts/test-cloud-run-helpers.mjs"
- "packages/scripts/test-cloud-run-flush.self-test.mjs"
- "packages/scripts/test-cloud-run-helpers.self-test.mjs"
- "package.json"
- "bun.lock"
- ".github/workflows/cloud-tests.yml"
- ".github/actions/cloud-setup-test-env/**"
push:
branches: [develop, main]
paths:
- "packages/cloud/api/**"
- "packages/cloud/shared/**"
- "packages/cloud/sdk/**"
- "packages/cloud/services/**"
- "packages/cloud/routing/**"
- "packages/cloud/infra/**"
- "packages/scripts/cloud/**"
- "packages/scripts/test-cloud-run.mjs"
- "packages/scripts/test-cloud-run-helpers.mjs"
- "packages/scripts/test-cloud-run-flush.self-test.mjs"
- "packages/scripts/test-cloud-run-helpers.self-test.mjs"
- "package.json"
- "bun.lock"
- ".github/workflows/cloud-tests.yml"
- ".github/actions/cloud-setup-test-env/**"
# CI fan-out fix: the PR-scoped concurrency from #14069 fell back to
# github.run_id on push, so every develop push got a unique group and
# nothing ever superseded. A 37-merge wave queued thousands of push runs
# and starved the self-hosted fleet. Fall back to github.ref (all develop
# pushes share a group) and cancel superseded runs on push too. PR behavior
# is unchanged (pull_request.number still wins the key); merge_group and
# schedule events are not in the cancel set, so the merge queue and nightly
# runs always complete.
concurrency:
group: cloud-tests-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }}
env:
DATABASE_URL: postgresql://postgres@127.0.0.1:5432/postgres
TEST_DATABASE_URL: postgresql://postgres@127.0.0.1:5432/postgres
CACHE_BACKEND: wadis
CACHE_ENABLED: "false"
NODE_OPTIONS: --max-old-space-size=4096
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
lint-and-types:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 25
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
- uses: ./.github/actions/cloud-setup-test-env
- name: Lint and typecheck cloud workspaces
run: bun run verify:cloud
- name: Tenant-scope gate (no unscoped pk-only reads of apps data-plane) (#9853)
run: bun run --cwd packages/cloud/shared check:tenant-scope
# NOTE: the Worker bundle stub-drift gate (#14422) runs inside
# `lint-and-types` above — packages/cloud/api chains `check:worker-bundle`
# (wrangler deploy --dry-run) into its `typecheck`, so `verify:cloud` covers
# it. Do not re-add a standalone dry-run job: #14474 + #14504 briefly ran it
# twice per PR on the saturated runner fleet.
unit-tests:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
# Cloud unit tests can run past 25m on saturated self-hosted runners while still passing.
timeout-minutes: 40
env:
NODE_ENV: test
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
- uses: ./.github/actions/cloud-setup-test-env
# Guards the batch runner's output-flush contract: a failing batch must
# surface its diagnostic instead of being swallowed by a process.exit()
# that truncates buffered stdout (the #16062 silent exit-1 regression).
- name: Batch runner flush-integrity self-test
run: node packages/scripts/test-cloud-run-flush.self-test.mjs
- name: Batch runner helper self-test
run: node packages/scripts/test-cloud-run-helpers.self-test.mjs
- name: Run cloud unit tests
run: bun run test:cloud
# The bun `test:cloud` lane skips suites that need the Vitest-only module
# mock API (`vi.mock` + `vi.importActual`) — see `SUPPORTS_VITEST_MOCK_API`.
# Run those under Vitest here so the direct-wallet payer/state-machine proof
# (33 tests, in-process PGlite) actually executes instead of vacuously
# skipping. Uses pglite://memory — no DB service required.
- name: Run cloud vitest-only suites
run: bun run --cwd packages/cloud/shared test:vitest
integration-tests:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 25
env:
NODE_ENV: test
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
- uses: ./.github/actions/cloud-setup-test-env
with:
setup-db: "true"
db-backend: "pglite"
- name: Run cloud integration tests
run: bun run test:cloud:integration
e2e-tests:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 35
needs: [lint-and-types]
env:
NODE_ENV: test
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
- uses: ./.github/actions/cloud-setup-test-env
with:
setup-db: "true"
- name: Run cloud e2e tests
run: bun run test:cloud:e2e
+100
View File
@@ -0,0 +1,100 @@
name: "CodeQL Advanced"
on:
push:
branches: ["main"]
pull_request:
branches: ["main"]
schedule:
- cron: "29 8 * * 6"
workflow_dispatch:
# Default to least privilege. Override per-job where needed.
# cancel superseded in-flight runs for the same PR to free hosted-runner
# capacity. only cancels pull_request runs; push runs on protected branches
# (main/develop) are never cancelled mid-flight.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
permissions:
contents: read
jobs:
analyze:
name: Analyze (${{ matrix.language }})
# WHY: whole-repo JS CodeQL can exhaust GitHub-hosted disk during the
# PolynomialReDoS dataflow query. Trusted events use the larger robot fleet;
# pull_request stays GitHub-hosted so forked code never runs on self-hosted.
runs-on: ${{ fromJSON(github.event_name == 'pull_request' && '["ubuntu-latest"]' || '["self-hosted","Linux","X64","hetzner-robot"]') }}
timeout-minutes: 360
permissions:
# required for all workflows
security-events: write
# required to fetch internal or private CodeQL packs
packages: read
# only required for workflows in private repositories
actions: read
contents: read
strategy:
fail-fast: false
matrix:
include:
- language: javascript-typescript
build-mode: none
steps:
- name: Checkout repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Report disk before CodeQL
shell: bash
run: |
df -h
du -sh "$GITHUB_WORKSPACE" 2>/dev/null || true
- name: Free GitHub-hosted runner disk
if: runner.environment == 'github-hosted'
shell: bash
run: |
# WHY: this preserves the full CodeQL query suite while giving the
# hosted fallback enough headroom for CodeQL's temporary databases.
sudo rm -rf \
/usr/local/lib/android \
/usr/share/dotnet \
/opt/ghc \
/opt/hostedtoolcache/go \
/opt/hostedtoolcache/Python \
/opt/hostedtoolcache/Ruby || true
df -h
- name: Initialize CodeQL
uses: github/codeql-action/init@99df26d4f13ea111d4ec1a7dddef6063f76b97e9
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
config-file: ./.github/codeql/codeql-config.yml
# WHY: the paths-ignore config shrinks the JS/TS database ~36x (~107MB), so
# the memory pressure behind exit-137 is resolved at the source. threads can
# return toward the machine's 12-core capacity for fast analysis (holding the
# robot briefly instead of ~1h); ram stays a generous ceiling (~120GB of the
# 244GB box) purely as a safety net against an unmeasured pathological query.
ram: 120000
threads: 8
- if: matrix.build-mode == 'manual'
shell: bash
run: |
echo 'If you are using a "manual" build mode for one or more of the' \
'languages you are analyzing, replace this with the commands to build' \
'your code, for example:'
echo ' make basic-capabilities'
echo ' make release'
exit 1
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@99df26d4f13ea111d4ec1a7dddef6063f76b97e9
with:
category: "/language:${{matrix.language}}"
+181
View File
@@ -0,0 +1,181 @@
name: coverage-gate
# SOC2 CC4.1 — track unit-test coverage on changed files. ENFORCED (issue #9943):
# a PR that changes source under packages/plugins/apps must change at least one
# unit test that can emit coverage. Every surviving changed executable source
# must appear in LCOV and clear the line-coverage floor. A source-aware
# classifier strips TypeScript-only syntax before excluding modules with no
# runtime code; declarations and deleted files are not targets. LCOV LF:0 never
# proves type-only status. Source covered only by e2e/live lanes still needs a
# focused unit test in this gate. Docs-only PRs do not spend coverage time.
# Standalone Node self-tests are excluded only when listed in the manifest that
# the Develop PR Test Integrity job executes on every PR.
# A test-only PR (changed tests, no changed source) executes its changed tests —
# gating the run steps on changed *source* let such PRs go vacuously green with
# the new test never running (#15849); the per-file floor simply has no source
# to enforce against.
#
# The per-package coverage floors live in the shared base vitest config
# (packages/test/vitest/default.config.ts, sourced from coverage-policy.mjs) and
# apply on a `--coverage` run. This changed-files gate is the second layer: it
# enforces a per-changed-file floor (`threshold` below), set to a ratchetable 50%
# — raise it toward the long-term 70% target as suites fill in. See issue #9943
# (the umbrella) and #10104 for the broader test-gating audit.
on:
pull_request:
branches: [develop, main]
paths:
- "packages/**"
- "plugins/**"
- "apps/**"
# cancel superseded in-flight runs for the same PR to free hosted-runner
# capacity. only cancels pull_request runs; push runs on protected branches
# (main/develop) are never cancelled mid-flight.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
permissions:
contents: read
env:
NODE_VERSION: "24.15.0"
jobs:
coverage:
name: coverage on changed files
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- name: Checkout
# actions/checkout@v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 0
- name: Setup Node.js for source classification
# The classifier is Node-based and runs before optional Bun workspace
# setup, including for JSON-only PRs on runners without system Node.
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Determine changed files
id: changed
# Three-dot diff (merge-base..HEAD) so a branch trailing develop does not
# count develop-side files it never touched. Logic lives in the script so
# its path/diff boundary rules are regression-tested (issue #15845).
run: |
bash scripts/security/coverage-changed-files.sh \
"${{ github.event.pull_request.base.sha }}" \
"${{ github.event.pull_request.head.sha }}" >> "$GITHUB_OUTPUT"
- name: Require changed tests for changed source
if: steps.changed.outputs.files != '' && steps.changed.outputs.bun_tests == '' && steps.changed.outputs.vitest_tests == '' && steps.changed.outputs.node_tests == ''
run: |
echo "Changed source files require at least one changed Bun or Vitest unit test:"
printf '%s\n' "${{ steps.changed.outputs.files }}"
exit 1
- name: Setup Bun workspace
if: steps.changed.outputs.bun_tests != '' || steps.changed.outputs.vitest_tests != ''
# Keep dependency setup behind the changed-test check so docs-only and
# test-free source PRs either skip or fail before spending setup time.
# Not gated on changed source: a test-only PR (files == '') must still
# execute its changed tests, or the suite goes vacuously green (#15849).
uses: ./.github/actions/setup-bun-workspace
with:
install-native-deps: "false"
install-protoc: "false"
setup-python: "false"
- name: Run changed registered Node self-tests
if: steps.changed.outputs.node_tests != ''
run: |
while IFS= read -r self_test; do
[ -n "$self_test" ] || continue
node "$self_test"
done <<'EOF'
${{ steps.changed.outputs.node_tests }}
EOF
- name: Run changed Bun tests with coverage
if: steps.changed.outputs.bun_tests != ''
run: |
cat > "$RUNNER_TEMP/changed-tests.txt" <<'EOF'
${{ steps.changed.outputs.bun_tests }}
EOF
mapfile -t changed_tests < <(grep -v '^$' "$RUNNER_TEMP/changed-tests.txt" || true)
if [ "${#changed_tests[@]}" -eq 0 ]; then
echo "no changed Bun-native test files; skipping coverage run"
exit 0
fi
shared_tests=()
process_isolated_tests=()
for test_file in "${changed_tests[@]}"; do
case "$test_file" in
packages/cloud/api/v1/voice/session/__tests__/harness-real-server.test.ts|packages/tools/voice-evidence-harness/src/cli-run.test.ts)
process_isolated_tests+=("$test_file")
;;
*) shared_tests+=("$test_file") ;;
esac
done
# This lane runs before workspace builds, so package defaults point at
# absent dist files; source exports keep changed tests self-contained.
if [ "${#shared_tests[@]}" -gt 0 ]; then
bun test --conditions=eliza-source "${shared_tests[@]}" --coverage --coverage-reporter=lcov --coverage-dir=coverage/bun/shared
fi
# Bun module mocks remain process-global despite `--isolate`. Tests
# listed here run in fresh processes and emit separate LCOV files;
# the gate below already aggregates every coverage/**/lcov.info.
for index in "${!process_isolated_tests[@]}"; do
ELIZA_PROCESS_ISOLATED_TEST=1 bun test --conditions=eliza-source "${process_isolated_tests[$index]}" --coverage --coverage-reporter=lcov --coverage-dir="coverage/bun/isolated-$index"
done
env:
# Tests must produce LCOV under this coverage root; the enforced gate
# recursively aggregates process-isolated and shared outputs.
BUN_COVERAGE_DIR: coverage/bun
- name: Build core for changed Vitest tests
if: steps.changed.outputs.vitest_tests != ''
# Vitest package configs may exercise source graphs that still depend on
# the standard built core/logger/contracts/cloud-routing entrypoints.
run: bun run --cwd packages/core build
- name: Run changed Vitest tests with coverage
if: steps.changed.outputs.vitest_tests != ''
run: |
cat > "$RUNNER_TEMP/changed-vitest-tests.txt" <<'EOF'
${{ steps.changed.outputs.vitest_tests }}
EOF
mapfile -t changed_tests < <(grep -v '^$' "$RUNNER_TEMP/changed-vitest-tests.txt" || true)
if [ "${#changed_tests[@]}" -eq 0 ]; then
echo "no changed Vitest test files; skipping coverage run"
exit 0
fi
node packages/scripts/run-changed-vitest-coverage.mjs "${changed_tests[@]}"
- name: Apply coverage gate (enforced)
# Runs whenever changed tests ran, including test-only PRs (files == '').
# With no changed source, executing the changed tests is the gate:
# Bun may not emit LCOV when it only ran tests, and there is no per-file
# floor to enforce (#15849).
if: steps.changed.outputs.bun_tests != '' || steps.changed.outputs.vitest_tests != '' || steps.changed.outputs.node_tests != ''
env:
COVERAGE_GATE_ENFORCE: "1" # enforced (issue #9943)
run: |
mapfile -t lcov_files < <(find coverage -name lcov.info -type f | sort)
if [ "${#lcov_files[@]}" -eq 0 ]; then
if [ -z "${{ steps.changed.outputs.files }}" ]; then
echo "changed tests ran; no changed source files require LCOV enforcement"
exit 0
fi
echo "changed tests ran but no coverage/lcov.info files were produced"
exit 1
fi
awk \
-v changed="${{ steps.changed.outputs.files }}" \
-v threshold=50 \
-f scripts/security/coverage-gate.awk \
"${lcov_files[@]}"
+311
View File
@@ -0,0 +1,311 @@
name: Deploy Apps Worker (Product 2)
# SSHes to the apps-control host, fetches the target branch, installs the
# systemd unit shipped at packages/scripts/cloud/admin/eliza-apps-worker.service,
# and restarts the eliza-apps-worker daemon (the Product-2 deploy daemon).
#
# This is the slim sibling of deploy-eliza-provisioning-worker.yml: it deploys
# ONLY the apps-worker — NO agent-router, and NO headscale wiring (the apps
# worker reaches the per-tenant Postgres over the apps PRIVATE network, never the
# agent mesh). The apps env (APPS_DEPLOY_ENABLED, APPS_TENANT_ADMIN_DSN,
# CONTAINERS_DOCKER_NODES, …) is written separately by arm-apps-daemon.yml — this
# workflow only ships code + the unit, keeping the two concerns split.
#
# First-time setup: bootstrap an apps-control VM on the apps PRIVATE network (so
# it can reach the tenant DB at 10.30.1.10) that runs NO untrusted containers,
# then set the env's ELIZA_APPS_WORKER_HOST + ELIZA_APPS_WORKER_SSH_KEY secrets.
on:
push:
branches: [develop, main]
paths:
- '.github/workflows/deploy-apps-worker.yml'
- 'packages/scripts/cloud/admin/daemons/apps-provisioning-worker.ts'
- 'packages/scripts/cloud/admin/daemons/provisioning-worker.ts'
- 'packages/scripts/cloud/admin/eliza-apps-worker.service'
- 'packages/cloud/shared/src/lib/services/provisioning-jobs.ts'
- 'packages/cloud/shared/src/lib/services/provisioning-job-types.ts'
- 'packages/cloud/shared/src/lib/services/apps-deploy-backend.ts'
- 'packages/cloud/shared/src/lib/services/tenant-db/**'
- 'packages/cloud/shared/src/lib/services/app-deploy-runner.ts'
- 'packages/cloud/sdk/**'
- 'packages/core/**'
- 'plugins/plugin-sql/**'
workflow_dispatch:
inputs:
environment:
description: 'Target environment (host + secrets resolved from this)'
required: true
default: 'staging'
type: choice
options:
- staging
- production
concurrency:
# Per-commit group (mirrors cloud-cf-deploy #10802/#10973). A SHARED
# deploy-apps-worker-<env> group with cancel-in-progress:false makes GitHub
# keep only the latest PENDING run per group and silently cancel the older
# pending ones (no annotation) — so a hot develop push/dispatch stream
# produces a chain of cancelled runs and zero completed deploys
# (elizaOS/eliza#10839). Keying on github.sha too gives each commit its own
# group so no run is ever silently superseded. Jobs run on ubuntu-latest
# (no runner-level serialization), so a host-side flock in the SSH scripts
# serializes concurrent deploys against the shared apps-control host; each
# run deploys the branch tip at fetch time, so the last to execute lands the
# newest code. Env stays in the key so staging and production never block
# each other.
group: deploy-apps-worker-${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production' || 'staging') }}-${{ github.sha }}
cancel-in-progress: false
permissions:
contents: read
jobs:
determine-env:
name: Determine environment
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
outputs:
environment: ${{ steps.env.outputs.environment }}
branch: ${{ steps.env.outputs.branch }}
steps:
- id: env
run: |
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
env="${{ github.event.inputs.environment }}"
elif [ "${{ github.ref }}" = "refs/heads/main" ]; then
env="production"
else
env="staging"
fi
branch=$([ "$env" = "production" ] && echo "main" || echo "develop")
echo "environment=$env" >> "$GITHUB_OUTPUT"
echo "branch=$branch" >> "$GITHUB_OUTPUT"
echo "Resolved: environment=$env branch=$branch"
deploy:
name: Deploy apps worker to apps-control host (${{ needs.determine-env.outputs.environment }})
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
needs: determine-env
environment: ${{ needs.determine-env.outputs.environment }}
timeout-minutes: 15
env:
SYSTEMD_UNIT: eliza-apps-worker.service
DEPLOY_HOST: ${{ secrets.ELIZA_APPS_WORKER_HOST }}
DEPLOY_SSH_KEY: ${{ secrets.ELIZA_APPS_WORKER_SSH_KEY }}
DEPLOY_BRANCH: ${{ needs.determine-env.outputs.branch }}
steps:
- name: Check deploy configuration
id: deploy_config
run: |
if [ -z "$DEPLOY_HOST" ] || [ -z "$DEPLOY_SSH_KEY" ]; then
echo "configured=false" >> "$GITHUB_OUTPUT"
echo "::warning::Missing ELIZA_APPS_WORKER_HOST or ELIZA_APPS_WORKER_SSH_KEY; skipping apps-worker deploy."
{
echo "### Apps worker deploy skipped"
echo ""
echo "Missing \`ELIZA_APPS_WORKER_HOST\` or \`ELIZA_APPS_WORKER_SSH_KEY\` for this environment."
echo "Bootstrap an apps-control VM on the apps private network and set those secrets first."
} >> "$GITHUB_STEP_SUMMARY"
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
echo "::error::Manual deployment requires ELIZA_APPS_WORKER_HOST and ELIZA_APPS_WORKER_SSH_KEY."
exit 1
fi
exit 0
fi
echo "configured=true" >> "$GITHUB_OUTPUT"
- name: Ensure host prereqs (Node 24, swap, bunx symlink)
if: steps.deploy_config.outputs.configured == 'true'
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2
with:
host: ${{ env.DEPLOY_HOST }}
username: deploy
key: ${{ env.DEPLOY_SSH_KEY }}
command_timeout: 5m
script: |
set -euo pipefail
if ! command -v node >/dev/null 2>&1; then
echo "[host-prereqs] installing Node 24"
curl -fsSL https://deb.nodesource.com/setup_24.x | sudo bash -
sudo apt-get install -y nodejs
fi
if [ ! -f /swapfile ]; then
echo "[host-prereqs] adding 8GB swap"
sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
grep -q '^/swapfile ' /etc/fstab || echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab >/dev/null
fi
if [ ! -e /home/deploy/.bun/bin/bunx ] && [ -e /home/deploy/.bun/bin/bun ]; then
echo "[host-prereqs] creating bunx symlink"
ln -sf bun /home/deploy/.bun/bin/bunx
fi
- name: Deploy and restart apps worker
if: steps.deploy_config.outputs.configured == 'true'
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2
with:
host: ${{ env.DEPLOY_HOST }}
username: deploy
key: ${{ env.DEPLOY_SSH_KEY }}
command_timeout: 10m
envs: DEPLOY_BRANCH,SYSTEMD_UNIT
script: |
set -euo pipefail
# Serialize deploys on the host: per-SHA concurrency groups mean
# overlapping runs are expected; without this they would interleave
# git clean/bun install/systemctl restart on the same checkout.
exec 9>/tmp/eliza-apps-worker-deploy.lock
# 540s keeps the lock wait under this step's 10m command_timeout so
# a starved waiter fails with THIS message, not an opaque SSH kill.
flock -w 540 9 || { echo "::error::another apps-worker deploy holds the host lock after 9m"; exit 1; }
echo "=== Deploying eliza-apps-worker (branch: $DEPLOY_BRANCH) ==="
cd /opt/eliza
export GIT_CONFIG_COUNT=1
export GIT_CONFIG_KEY_0=safe.directory
export GIT_CONFIG_VALUE_0=/opt/eliza
rm -f .git/index.lock
sudo chown -R deploy:deploy .git
git reset --hard HEAD 2>/dev/null || true
git clean -fdx \
-e .env \
-e .env.local \
-e '.env.*' \
-e cloud/.env.local \
-e 'cloud/.env.*' \
-e node_modules
find packages -type f \( -name '*.d.ts' -o -name '*.d.ts.map' \) \
! -path '*/node_modules/*' -delete 2>/dev/null || true
rm -f bun.lock
git -c fetch.recurseSubmodules=no fetch --no-recurse-submodules \
origin "+$DEPLOY_BRANCH:refs/remotes/origin/$DEPLOY_BRANCH"
git -c submodule.recurse=false checkout --no-recurse-submodules \
-B "$DEPLOY_BRANCH" "origin/$DEPLOY_BRANCH"
# The `rm -f bun.lock` above deletes a TRACKED file; when bun.lock
# is identical between the previous HEAD and the new tip, checkout
# carries that deletion over as a local change and `bun install`
# then resolves the whole tree fresh from package.json ranges —
# newest-in-range @ai-sdk/* against the overridden provider-utils
# pin took the worker down at boot (`does not provide an export
# named 'secureJsonParse'`). Always restore the tracked lockfile so
# installs are lockfile-pinned and deterministic.
git checkout "origin/$DEPLOY_BRANCH" -- bun.lock
if ! command -v bun >/dev/null 2>&1; then
curl -fsSL https://bun.sh/install | bash -s "canary"
export BUN_INSTALL="$HOME/.bun"
export PATH="$BUN_INSTALL/bin:$PATH"
fi
cd /opt/eliza
for attempt in 1 2 3; do
if bun install --no-save --ignore-scripts; then
break
fi
if [ "$attempt" -eq 3 ]; then exit 1; fi
echo "bun install attempt $attempt failed; retrying in 10s..."
sleep 10
done
# The daemon runs under Node/tsx, which resolves linked workspace
# packages through their built `node` exports — refresh dist files.
bun run build:core
mkdir -p plugins/plugin-sql/node_modules/@elizaos
rm -rf plugins/plugin-sql/node_modules/@elizaos/core
ln -s ../../../../packages/core plugins/plugin-sql/node_modules/@elizaos/core
bun run --cwd plugins/plugin-sql build
# packages/scripts has no package.json, so bun's workspace graph
# doesn't link @elizaos/cloud-shared for the daemon — create it.
mkdir -p node_modules/@elizaos
ln -sfn ../../packages/cloud/shared node_modules/@elizaos/cloud-shared
sudo install -m 0644 \
packages/scripts/cloud/admin/eliza-apps-worker.service \
"/etc/systemd/system/$SYSTEMD_UNIT"
sudo systemctl daemon-reload
sudo systemctl enable "$SYSTEMD_UNIT"
sudo systemctl restart "$SYSTEMD_UNIT"
echo "=== Restart issued for apps worker ==="
- name: Health check
if: steps.deploy_config.outputs.configured == 'true'
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2
with:
host: ${{ env.DEPLOY_HOST }}
username: deploy
key: ${{ env.DEPLOY_SSH_KEY }}
command_timeout: 5m
envs: SYSTEMD_UNIT
script: |
set -euo pipefail
# Hold the same host lock as the deploy step so a concurrent run's
# restart can never flip the unit mid-check (false red). 240s wait
# stays under this step's 5m command_timeout.
exec 9>/tmp/eliza-apps-worker-deploy.lock
flock -w 240 9 || { echo "::error::another apps-worker deploy holds the host lock after 4m"; exit 1; }
# The apps worker has no HTTP endpoint; health = active + stable +
# no fatal/restart-loop in the journal since deploy.
HEALTH_SINCE_TS="$(date -u '+%Y-%m-%d %H:%M:%S')"
STABLE_THRESHOLD_SEC=20
FATAL_LOG_PATTERN="\[apps-worker\] (fatal|unhandled rejection)|node:internal/.*Error:|Error \[ERR_|^[A-Za-z]+Error:"
for attempt in $(seq 1 18); do
if sudo systemctl is-active --quiet "$SYSTEMD_UNIT"; then
JOURNAL=$(sudo journalctl -u "$SYSTEMD_UNIT" --since "$HEALTH_SINCE_TS" --no-pager 2>/dev/null || true)
if echo "$JOURNAL" | grep -qE "$FATAL_LOG_PATTERN"; then
echo "::error::Apps worker logged a fatal error since deploy."
echo "$JOURNAL" | tail -n 50
exit 1
fi
UPTIME_SEC=$(systemctl show "$SYSTEMD_UNIT" --property=ActiveEnterTimestampMonotonic --value | awk '{ print int($1 / 1e6) }')
NOW_SEC=$(awk '{print int($1)}' /proc/uptime)
AGE=$(( NOW_SEC - UPTIME_SEC ))
if [ "$AGE" -ge "$STABLE_THRESHOLD_SEC" ]; then
echo "Apps worker active and stable (${AGE}s) on attempt $attempt."
exit 0
fi
echo "Health check attempt $attempt/18: active but only ${AGE}s old, waiting for stability..."
else
echo "Health check attempt $attempt/18: not active yet."
fi
sleep 5
done
echo "::error::$SYSTEMD_UNIT failed to become healthy within 90s."
sudo systemctl status "$SYSTEMD_UNIT" --no-pager || true
sudo journalctl -u "$SYSTEMD_UNIT" -n 200 --no-pager || true
exit 1
- name: Notify Discord (Success)
if: success() && steps.deploy_config.outputs.configured == 'true'
uses: sarisia/actions-status-discord@eb045afee445dc055c18d3d90bd0f244fd062708
with:
webhook: ${{ secrets.DISCORD_WEBHOOK }}
ack_no_webhook: true
title: "🚀 Eliza Apps Worker Deployed (Product 2)"
nodetail: true
description: |
Branch: ${{ needs.determine-env.outputs.branch }}
Commit: ${{ github.sha }}
color: 0x00ff00
- name: Notify Discord (Failure)
if: failure() && steps.deploy_config.outputs.configured == 'true'
uses: sarisia/actions-status-discord@eb045afee445dc055c18d3d90bd0f244fd062708
with:
webhook: ${{ secrets.DISCORD_WEBHOOK }}
ack_no_webhook: true
title: "❌ Eliza Apps Worker Deploy Failed"
nodetail: true
description: |
Branch: ${{ needs.determine-env.outputs.branch }}
Commit: ${{ github.sha }}
color: 0xff0000
@@ -0,0 +1,550 @@
name: Deploy Eliza Provisioning Worker
# SSHes to the Hetzner provisioning host and deploys one immutable commit,
# installs the systemd unit shipped at packages/scripts/cloud/admin/eliza-provisioning-worker.service,
# and restarts the eliza-provisioning-worker daemon.
#
# First-time setup:
# HCLOUD_TOKEN=... bun run packages/scripts/cloud/admin/bootstrap-provisioning-worker-host.mjs
on:
push:
branches: [develop, main]
paths:
- '.github/workflows/deploy-eliza-provisioning-worker.yml'
- 'packages/scripts/cloud/admin/daemons/provisioning-worker.ts'
- 'packages/scripts/cloud/admin/daemons/agent-router.ts'
- 'packages/scripts/cloud/admin/eliza-provisioning-worker.service'
- 'packages/scripts/cloud/admin/eliza-agent-router.service'
- 'packages/scripts/cloud/admin/ensure-generated-keywords.sh'
- 'packages/cloud/shared/src/lib/services/provisioning-jobs.ts'
- 'packages/cloud/shared/src/lib/services/docker-sandbox-provider.ts'
- 'packages/cloud/shared/src/lib/services/eliza-sandbox.ts'
- 'packages/cloud/shared/src/db/repositories/agent-sandboxes.ts'
- 'packages/cloud/shared/**'
- 'packages/cloud/sdk/**'
- 'packages/shared/**'
- 'packages/core/**'
- 'plugins/plugin-sql/**'
workflow_dispatch:
inputs:
environment:
description: 'Target environment (host + secrets resolved from this)'
required: true
default: 'staging'
type: choice
options:
- staging
- production
deployment_sha:
description: 'Optional exact repository commit for protected staging acceptance'
required: false
type: string
# Per-env concurrency: a develop push (staging) and a main push (production)
# can deploy in parallel without one cancelling the other.
concurrency:
group: deploy-eliza-provisioning-worker-${{ github.event.inputs.environment || (github.ref == 'refs/heads/main' && 'production' || 'staging') }}
cancel-in-progress: false
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
determine-env:
name: Determine environment
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
outputs:
environment: ${{ steps.env.outputs.environment }}
branch: ${{ steps.env.outputs.branch }}
deployment_sha: ${{ steps.snapshot.outputs.deployment_sha }}
steps:
- id: env
run: |
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
env="${{ github.event.inputs.environment }}"
elif [ "${{ github.ref }}" = "refs/heads/main" ]; then
env="production"
else
env="staging"
fi
branch=$([ "$env" = "production" ] && echo "main" || echo "develop")
echo "environment=$env" >> "$GITHUB_OUTPUT"
echo "branch=$branch" >> "$GITHUB_OUTPUT"
echo "Resolved: environment=$env branch=$branch"
- id: snapshot
env:
BRANCH: ${{ steps.env.outputs.branch }}
PUSH_SHA: ${{ github.sha }}
REQUESTED_SHA: ${{ github.event.inputs.deployment_sha }}
TARGET_ENVIRONMENT: ${{ steps.env.outputs.environment }}
run: |
set -euo pipefail
if [ "${{ github.event_name }}" = "push" ]; then
deployment_sha="$PUSH_SHA"
elif [ -n "$REQUESTED_SHA" ]; then
[ "$TARGET_ENVIRONMENT" = "staging" ] || {
echo "::error::An exact deployment_sha is permitted only for protected staging deployments"
exit 1
}
[[ "$REQUESTED_SHA" =~ ^[0-9a-f]{40}$ ]] || {
echo "::error::deployment_sha must be a lowercase 40-hex commit"
exit 1
}
verify_dir=$(mktemp -d)
trap 'rm -rf "$verify_dir"' EXIT
git -C "$verify_dir" init --quiet
git -C "$verify_dir" fetch --quiet --no-tags --depth=1 \
"https://github.com/${GITHUB_REPOSITORY}.git" "$REQUESTED_SHA"
deployment_sha=$(git -C "$verify_dir" rev-parse FETCH_HEAD)
[ "$deployment_sha" = "$REQUESTED_SHA" ] || {
echo "::error::Fetched commit does not match requested deployment_sha"
exit 1
}
else
deployment_sha=$(git ls-remote "https://github.com/${GITHUB_REPOSITORY}.git" "refs/heads/$BRANCH" | awk 'NR == 1 { print $1 }')
fi
[[ "$deployment_sha" =~ ^[0-9a-f]{40}$ ]] || { echo "::error::Could not resolve immutable deployment SHA for $BRANCH"; exit 1; }
echo "deployment_sha=$deployment_sha" >> "$GITHUB_OUTPUT"
echo "Resolved immutable deployment snapshot: $deployment_sha"
deploy:
name: Deploy worker to Hetzner host (${{ needs.determine-env.outputs.environment }} @ ${{ needs.determine-env.outputs.deployment_sha }})
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
needs: determine-env
environment: ${{ needs.determine-env.outputs.environment }}
timeout-minutes: 15
env:
SYSTEMD_UNIT: eliza-provisioning-worker.service
DEPLOY_HOST: ${{ secrets.ELIZA_PROVISIONING_HOST }}
DEPLOY_SSH_KEY: ${{ secrets.ELIZA_PROVISIONING_SSH_KEY }}
DEPLOY_BRANCH: ${{ needs.determine-env.outputs.branch }}
DEPLOY_SHA: ${{ needs.determine-env.outputs.deployment_sha }}
# Headscale wiring for the provisioning worker (the CONSUMER of the
# control plane). These are reconciled into /opt/eliza/cloud/.env.local
# on every deploy so the box can never silently drift to a stale,
# hand-edited headscale URL/key again. Reuse the canonical GH names
# already established by arm-headscale-control-plane.yml.
HEADSCALE_API_URL: ${{ vars.HEADSCALE_API_URL }}
HEADSCALE_PUBLIC_URL: ${{ vars.HEADSCALE_PUBLIC_URL }}
HEADSCALE_API_KEY: ${{ secrets.HEADSCALE_API_KEY }}
# Base64 private key used by the daemon to reach dedicated-agent nodes.
# Keep this in the protected environment instead of relying on a
# hand-copied file that can drift or disappear when the CP host is rebuilt.
CONTAINERS_SSH_KEY: ${{ secrets.CONTAINERS_SSH_KEY }}
# Sandbox-registry Redis (TCP redis:// proxy, sandbox-reachable — never a
# *.railway.internal host) so provisioned sandboxes self-register and
# Discord/Telegram inbound routing works (#8621 / #8756). Reconciled into
# /opt/eliza/cloud/.env.local the same way as HEADSCALE_*; when the secret
# is unset the reconcile loop skips it and preserves any hand-set value,
# so this is a no-op until the operator sets the GH secret.
SANDBOX_REGISTRY_REDIS_URL: ${{ secrets.SANDBOX_REGISTRY_REDIS_URL }}
# Cloud database DSN for the provisioning worker + agent router. Same
# fallback chain as the migrator jobs (cloud-cf-deploy.yml /
# cloud-deploy-backend.yml) so the CP daemon always resolves the exact
# DB the Worker schema is migrated against. Before this, DATABASE_URL
# only lived hand-edited on the box: when staging moved Neon → Railway
# the CP silently kept polling the abandoned Neon DB for 3 weeks
# (#15160). Reconciled into /opt/eliza/cloud/.env.local with the same
# skip-when-unset semantics as HEADSCALE_* — an environment without any
# of these secrets keeps its current hand-set value.
DATABASE_URL: ${{ secrets.DATABASE_URL || secrets.RAILWAY_DATABASE_URL || secrets.NEON_DATABASE_URL }}
# Field-encryption root secret used to unwrap per-org DEKs for encrypted
# agent environment variables. Must match the Cloudflare Worker secret:
# the Worker writes encrypted env vars, and this provisioning worker
# decrypts them while creating the agent container (#15385). Skip-empty
# reconcile semantics below preserve hand-set values until the secret is
# wired for an environment.
SECRETS_MASTER_KEY: ${{ secrets.SECRETS_MASTER_KEY }}
# Operator pin for the canonical managed-agent image
# (containersEnv.defaultAgentImageOverride; falls back to
# ghcr.io/elizaos/eliza:stable when unset everywhere). This pin decides
# which image every NEW provision and warm-pool replenish boots — it is
# exactly the knob the #15228 crash-loop rollout needed. Before this it
# lived ONLY hand-set on the CP box (the same drift class as
# DATABASE_URL/#15160, tracked in the #15401 ledger): a CP re-arm lost
# it silently. Set the GitHub environment VARIABLE (not secret — an
# image ref is auditable config) to roll the fleet pin via deploy;
# leave it unset to preserve whatever is hand-set on the box.
ELIZA_AGENT_IMAGE: ${{ vars.ELIZA_AGENT_IMAGE }}
# The edge Worker forwards the original per-agent host to this daemon.
# Keep the daemon's parser on the same environment-specific suffix or a
# staging host (`<uuid>.staging.elizacloud.ai`) is rejected as malformed.
ELIZA_CLOUD_AGENT_BASE_DOMAIN: ${{ needs.determine-env.outputs.environment == 'production' && 'elizacloud.ai' || 'staging.elizacloud.ai' }}
steps:
- name: Check deploy configuration
id: deploy_config
run: |
if [ -z "$DEPLOY_HOST" ] || [ -z "$DEPLOY_SSH_KEY" ]; then
echo "configured=false" >> "$GITHUB_OUTPUT"
echo "::warning::Missing ELIZA_PROVISIONING_HOST or ELIZA_PROVISIONING_SSH_KEY; skipping provisioning-worker deploy."
{
echo "### Provisioning worker deploy skipped"
echo ""
echo "Missing \`ELIZA_PROVISIONING_HOST\` or \`ELIZA_PROVISIONING_SSH_KEY\` for GitHub environment \`production\`."
echo ""
echo "Bootstrap the Hetzner host and secrets with:"
echo ""
echo "\`\`\`bash"
echo "HCLOUD_TOKEN=... bun run packages/scripts/cloud/admin/bootstrap-provisioning-worker-host.mjs"
echo "\`\`\`"
} >> "$GITHUB_STEP_SUMMARY"
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
echo "::error::Manual deployment requires ELIZA_PROVISIONING_HOST and ELIZA_PROVISIONING_SSH_KEY."
exit 1
fi
exit 0
fi
echo "configured=true" >> "$GITHUB_OUTPUT"
- name: Ensure host prereqs (Node 24, swap, bunx symlink)
if: steps.deploy_config.outputs.configured == 'true'
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2
with:
host: ${{ env.DEPLOY_HOST }}
username: deploy
key: ${{ env.DEPLOY_SSH_KEY }}
command_timeout: 5m
script: |
set -euo pipefail
# Idempotent safety net for hosts provisioned BEFORE the cloud-init
# template that pre-installs these. Each block is a no-op if already
# set up — runs on every deploy so any host drift self-heals.
# Node 24: the systemd unit calls `tsx` which spawns a Node process.
# New cloud-init template installs it via NodeSource; older hosts
# (eliza-staging-1 from the first TF apply) lack it.
if ! command -v node >/dev/null 2>&1; then
echo "[host-prereqs] installing Node 24"
curl -fsSL https://deb.nodesource.com/setup_24.x | sudo bash -
sudo apt-get install -y nodejs
fi
# Swap: smaller shapes (cpx22, cax11) OOM during `turbo run build`.
# 8 GB swap once and forever; harmless on bigger shapes.
if [ ! -f /swapfile ]; then
echo "[host-prereqs] adding 8GB swap"
sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
grep -q '^/swapfile ' /etc/fstab || echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab >/dev/null
fi
# bunx symlink: the canary `bun.sh/install` tarball stopped shipping
# bunx as a separate binary at some point; build scripts that call
# `bunx tsc` etc. then fail with `bun: command not found: bunx`.
# `bunx` is just `bun` reading argv[0] — a symlink is sufficient.
if [ ! -e /home/deploy/.bun/bin/bunx ] && [ -e /home/deploy/.bun/bin/bun ]; then
echo "[host-prereqs] creating bunx symlink"
ln -sf bun /home/deploy/.bun/bin/bunx
fi
- name: Deploy and restart worker
if: steps.deploy_config.outputs.configured == 'true'
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2
with:
host: ${{ env.DEPLOY_HOST }}
username: deploy
key: ${{ env.DEPLOY_SSH_KEY }}
command_timeout: 10m
envs: DEPLOY_BRANCH,DEPLOY_SHA,SYSTEMD_UNIT,HEADSCALE_API_URL,HEADSCALE_PUBLIC_URL,HEADSCALE_API_KEY,CONTAINERS_SSH_KEY,SANDBOX_REGISTRY_REDIS_URL,DATABASE_URL,SECRETS_MASTER_KEY,ELIZA_AGENT_IMAGE,ELIZA_CLOUD_AGENT_BASE_DOMAIN
script: |
set -euo pipefail
echo "=== Deploying eliza-provisioning-worker (branch: $DEPLOY_BRANCH sha: $DEPLOY_SHA) ==="
cd /opt/eliza
export GIT_CONFIG_COUNT=1
export GIT_CONFIG_KEY_0=safe.directory
export GIT_CONFIG_VALUE_0=/opt/eliza
rm -f .git/index.lock
# A prior operator or deploy may have run git/install/build as
# root, leaving either Git metadata or tracked source files owned
# by root. Git then cannot reset or check out the immutable target
# as `deploy`. The checkout is exclusively managed by this service,
# so normalize the whole tree before any mutating Git command.
sudo chown -R deploy:deploy /opt/eliza
# Discard local drift, preserve env files. Quote globs so the
# remote shell doesn't expand them before git sees the pattern,
# and fail loud if clean errors out — silently swallowing it is
# what let stale build artifacts survive and block checkout below.
git reset --hard HEAD
git clean -fdx \
-e .env \
-e .env.local \
-e '.env.*' \
-e cloud/.env.local \
-e 'cloud/.env.*' \
-e node_modules
# Build artifacts (*.d.ts / *.d.ts.map) sometimes land next to
# sources from a prior deploy; if the new HEAD tracks the same
# paths, `git checkout -B` aborts with "untracked working tree
# files would be overwritten". Sweep them before checkout.
find packages -type f \( -name '*.d.ts' -o -name '*.d.ts.map' \) \
! -path '*/node_modules/*' -delete
# bun.lock became tracked at 4ae092dcda; the host's previous deploy
# may have left an untracked copy from `bun install` that pre-dates
# the tracking change. `git clean -fdx` *should* remove it but
# occasionally races with bun's lockfile rewrite, so force a delete
# before the fetch + branch-checkout below.
rm -f bun.lock
# Use a refspec so a single-branch / shallow clone still creates
# the remote-tracking ref for $DEPLOY_BRANCH on first deploy.
#
# --no-recurse-submodules on both fetch + checkout: the provisioning
# worker daemon and agent router are pure Node/tsx code and never
# touch the llama.cpp / opencode submodules. If the
# deploy host has `fetch.recurseSubmodules=on-demand` (the default
# when submodules are present), recursing through a superproject
# commit whose submodule SHA was never pushed to the fork kills the
# whole deploy with `upload-pack: not our ref`. Skip submodule
# traversal entirely on this path.
git -c fetch.recurseSubmodules=no fetch --no-recurse-submodules origin "$DEPLOY_SHA"
git -c submodule.recurse=false checkout --no-recurse-submodules \
-B "$DEPLOY_BRANCH" "$DEPLOY_SHA"
test "$(git rev-parse HEAD)" = "$DEPLOY_SHA"
# The `rm -f bun.lock` above deletes a TRACKED file; when bun.lock
# is identical between the previous HEAD and the new tip, checkout
# carries that deletion over as a local change and `bun install`
# then resolves the whole tree fresh from package.json ranges —
# newest-in-range @ai-sdk/* against the overridden provider-utils
# pin took the worker down at boot (`does not provide an export
# named 'secureJsonParse'`). Always restore the tracked lockfile so
# installs are lockfile-pinned and deterministic.
git checkout "$DEPLOY_SHA" -- bun.lock
# git clean removes these intentionally ignored modules. Generate
# and assert them before install/build, while the healthy daemons
# are still running, so generator failure cannot cause downtime.
bash packages/scripts/cloud/admin/ensure-generated-keywords.sh
# Install the canary (Rust) Bun on first install.
if ! command -v bun >/dev/null 2>&1; then
curl -fsSL https://bun.sh/install | bash -s "canary"
export BUN_INSTALL="$HOME/.bun"
export PATH="$BUN_INSTALL/bin:$PATH"
fi
cd /opt/eliza
for attempt in 1 2 3; do
if bun install --no-save --ignore-scripts; then
break
fi
if [ "$attempt" -eq 3 ]; then exit 1; fi
echo "bun install attempt $attempt failed; retrying in 10s..."
sleep 10
done
# The daemons run under Node/tsx rather than Bun. Node resolves
# linked workspace packages through their built `node` exports, so
# refresh linked package dist files after installing dependencies.
#
# The `git clean -fdx` above wipes `.turbo`, so every deploy does a
# cold `build:core` (0 cached). Building the full graph includes
# `@elizaos/plugin-local-inference`, whose tsc/bundler step exceeds
# V8's default ~2 GB old-space and aborts the deploy with
# `FATAL ERROR: ... JavaScript heap out of memory` (exit 134). Raise
# the heap ceiling to match the repo's typecheck lane (8 GB).
export NODE_OPTIONS="${NODE_OPTIONS:-} --max-old-space-size=8192"
bun run build:core
mkdir -p plugins/plugin-sql/node_modules/@elizaos
rm -rf plugins/plugin-sql/node_modules/@elizaos/core
ln -s ../../../../packages/core plugins/plugin-sql/node_modules/@elizaos/core
bun run --cwd plugins/plugin-sql build
# Defensive workspace symlinks: `bun install` should create these
# automatically for any workspace package the root or a sibling
# depends on, but the daemon (in packages/scripts/, which has NO
# package.json and so is invisible to bun's workspace graph)
# imports `@elizaos/cloud-shared` at runtime. On a fresh host the
# symlink was missing — daemon then crashes with
# `Cannot find package '@elizaos/cloud-shared'`. Recreate it here
# idempotently after `bun install` is done.
mkdir -p node_modules/@elizaos
ln -sfn ../../packages/cloud/shared node_modules/@elizaos/cloud-shared
# Install/refresh both systemd units (provisioning worker + agent router).
sudo install -m 0644 \
packages/scripts/cloud/admin/eliza-provisioning-worker.service \
"/etc/systemd/system/$SYSTEMD_UNIT"
sudo install -m 0644 \
packages/scripts/cloud/admin/eliza-agent-router.service \
/etc/systemd/system/eliza-agent-router.service
sudo systemctl daemon-reload
sudo systemctl enable "$SYSTEMD_UNIT" eliza-agent-router.service
# Reconcile control-plane secrets (HEADSCALE_API_URL /
# HEADSCALE_PUBLIC_URL / HEADSCALE_API_KEY /
# SANDBOX_REGISTRY_REDIS_URL / DATABASE_URL / SECRETS_MASTER_KEY)
# into the systemd EnvironmentFile so CI is the single source of
# truth and the box self-heals on every deploy. The daemon +
# cloud-shared read these straight from process.env
# (headscale-client.ts, headscale-integration.ts,
# docker-sandbox-provider.ts, the DB client, and encrypted
# environment-variable decrypt), fed by
# EnvironmentFile=/opt/eliza/cloud/.env.local. The
# deploy's `git clean -e cloud/.env.local` preserves that file
# verbatim, so before this these keys were only ever hand-edited on
# the box and could drift to a stale headscale (the CP-box outage)
# or a stale database (#15160: the CP polled an abandoned Neon DB
# for 3 weeks after staging moved to Railway). Only rewrite a key
# when the CI value is non-empty — an unset GH var/secret must
# never blank a working box (a blank HEADSCALE_API_KEY makes
# docker-sandbox-provider throw "HEADSCALE_API_KEY is not
# configured" when a route is required).
ENV_FILE=/opt/eliza/cloud/.env.local
sudo touch "$ENV_FILE"
# Pin this control-plane daemon to the AGENT lane. Unset, a single
# daemon claims ALL job types (agent + apps); it would claim
# APP_DEPLOY / CONTAINER_* jobs it can't run (no apps backend wired
# here) and burn them down through retries. The dedicated apps
# daemon (apps-provisioning-worker.ts) owns the apps lane. Reconciled
# here so CI is the single source of truth and the box self-heals on
# every deploy (same mechanism as the HEADSCALE_* keys below). This
# value is constant — not a GH var — so it's reconciled directly,
# never blanked.
sudo sed -i "/^PROVISIONING_JOB_LANES=/d" "$ENV_FILE"
printf 'PROVISIONING_JOB_LANES=%s\n' "agent" | sudo tee -a "$ENV_FILE" >/dev/null
# Headscale is the required dedicated-agent ingress. Bridge-host
# fallback was a legacy escape hatch; if it remains hand-set on the
# box, docker-sandbox-provider deliberately disables VPN injection
# and fresh staging agents never receive a headscale_ip (#15347).
sudo sed -i "/^AGENT_ROUTER_ALLOW_BRIDGE_HOST_FALLBACK=/d" "$ENV_FILE"
for kv in \
"HEADSCALE_API_URL=$HEADSCALE_API_URL" \
"HEADSCALE_PUBLIC_URL=$HEADSCALE_PUBLIC_URL" \
"HEADSCALE_API_KEY=$HEADSCALE_API_KEY" \
"CONTAINERS_SSH_KEY=$CONTAINERS_SSH_KEY" \
"SANDBOX_REGISTRY_REDIS_URL=$SANDBOX_REGISTRY_REDIS_URL" \
"DATABASE_URL=$DATABASE_URL" \
"SECRETS_MASTER_KEY=$SECRETS_MASTER_KEY" \
"ELIZA_AGENT_IMAGE=$ELIZA_AGENT_IMAGE" \
"ELIZA_CLOUD_AGENT_BASE_DOMAIN=$ELIZA_CLOUD_AGENT_BASE_DOMAIN"; do
key="${kv%%=*}"
val="${kv#*=}"
[ -n "$val" ] || continue
sudo sed -i "/^${key}=/d" "$ENV_FILE"
printf '%s=%s\n' "$key" "$val" | sudo tee -a "$ENV_FILE" >/dev/null
done
sudo chmod 600 "$ENV_FILE"
sudo systemctl restart "$SYSTEMD_UNIT"
sudo systemctl restart eliza-agent-router.service
echo "=== Restart issued for both daemons ==="
- name: Health check
if: steps.deploy_config.outputs.configured == 'true'
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2
with:
host: ${{ env.DEPLOY_HOST }}
username: deploy
key: ${{ env.DEPLOY_SSH_KEY }}
command_timeout: 5m
envs: SYSTEMD_UNIT,ELIZA_CLOUD_AGENT_BASE_DOMAIN
script: |
set -euo pipefail
ENV_FILE=/opt/eliza/cloud/.env.local
ACTUAL_AGENT_BASE_DOMAIN=$(sudo awk -F= '$1=="ELIZA_CLOUD_AGENT_BASE_DOMAIN" { value=$2 } END { print value }' "$ENV_FILE")
if [ "$ACTUAL_AGENT_BASE_DOMAIN" != "$ELIZA_CLOUD_AGENT_BASE_DOMAIN" ]; then
echo "::error::Agent router base-domain drift: expected $ELIZA_CLOUD_AGENT_BASE_DOMAIN, found ${ACTUAL_AGENT_BASE_DOMAIN:-<missing>}."
exit 1
fi
# The worker logs to journald via systemd's stdout/stderr capture.
# Info-level logs are filtered unless VERBOSE_LOGGING=true, so we
# rely on systemd's own signals: process is active AND no fatal
# error / restart-loop in the journal since deploy.
HEALTH_SINCE_TS="$(date -u '+%Y-%m-%d %H:%M:%S')"
STABLE_THRESHOLD_SEC=20
FATAL_LOG_PATTERN="\[(provisioning-worker|agent-router)\] (fatal|unhandled rejection)|node:internal/.*Error:|Error \[ERR_|^[A-Za-z]+Error:"
for attempt in $(seq 1 18); do
if sudo systemctl is-active --quiet "$SYSTEMD_UNIT"; then
JOURNAL=$(sudo journalctl -u "$SYSTEMD_UNIT" --since "$HEALTH_SINCE_TS" --no-pager 2>/dev/null || true)
if echo "$JOURNAL" | grep -qE "$FATAL_LOG_PATTERN"; then
echo "::error::Worker logged a fatal error since deploy."
echo "$JOURNAL" | tail -n 50
exit 1
fi
# Active + uptime past the threshold = healthy. systemd's Restart=always
# means a crashing process would resurrect; an uptime > threshold
# therefore proves it stayed up across at least one poll cycle.
UPTIME_SEC=$(systemctl show "$SYSTEMD_UNIT" --property=ActiveEnterTimestampMonotonic --value | awk '{ print int($1 / 1e6) }')
NOW_SEC=$(awk '{print int($1)}' /proc/uptime)
AGE=$(( NOW_SEC - UPTIME_SEC ))
if [ "$AGE" -ge "$STABLE_THRESHOLD_SEC" ]; then
if ! sudo systemctl is-active --quiet eliza-agent-router.service; then
echo "Health check attempt $attempt/18: worker stable (${AGE}s), router not active yet."
sleep 5
continue
fi
ROUTER_JOURNAL=$(sudo journalctl -u eliza-agent-router.service --since "$HEALTH_SINCE_TS" --no-pager 2>/dev/null || true)
if echo "$ROUTER_JOURNAL" | grep -qE "$FATAL_LOG_PATTERN"; then
echo "::error::Agent router logged a fatal error since activation."
echo "$ROUTER_JOURNAL" | tail -n 50
exit 1
fi
ROUTER_UPTIME_SEC=$(systemctl show eliza-agent-router.service --property=ActiveEnterTimestampMonotonic --value | awk '{ print int($1 / 1e6) }')
ROUTER_AGE=$(( NOW_SEC - ROUTER_UPTIME_SEC ))
if [ "$ROUTER_AGE" -lt "$STABLE_THRESHOLD_SEC" ]; then
echo "Health check attempt $attempt/18: worker stable (${AGE}s), router only ${ROUTER_AGE}s old."
sleep 5
continue
fi
ROUTER_PORT=$(systemctl show eliza-agent-router.service --property=Environment --value | tr ' ' '\n' | awk -F= '$1=="AGENT_ROUTER_PORT" {print $2}')
: "${ROUTER_PORT:=3458}"
if curl -sf -m 3 "http://127.0.0.1:${ROUTER_PORT}/healthz" >/dev/null 2>&1; then
echo "Both daemons active and stable (worker ${AGE}s, router ${ROUTER_AGE}s, router /healthz OK) on attempt $attempt."
exit 0
fi
echo "Health check attempt $attempt/18: worker stable (${AGE}s), router ${ROUTER_AGE}s but /healthz not ready on port ${ROUTER_PORT}."
sleep 5
continue
fi
echo "Health check attempt $attempt/18: active but only ${AGE}s old, waiting for stability..."
else
echo "Health check attempt $attempt/18: not active yet."
fi
sleep 5
done
echo "::error::$SYSTEMD_UNIT failed to become healthy within 90s."
sudo systemctl status "$SYSTEMD_UNIT" --no-pager || true
sudo journalctl -u "$SYSTEMD_UNIT" -n 200 --no-pager || true
sudo systemctl status eliza-agent-router.service --no-pager || true
sudo journalctl -u eliza-agent-router.service -n 200 --no-pager || true
exit 1
- name: Notify Discord (Success)
if: success() && steps.deploy_config.outputs.configured == 'true'
uses: sarisia/actions-status-discord@eb045afee445dc055c18d3d90bd0f244fd062708
with:
webhook: ${{ secrets.DISCORD_WEBHOOK }}
ack_no_webhook: true
title: "🚀 Eliza Provisioning Worker Deployed"
nodetail: true
description: |
Branch: develop
Commit: ${{ github.sha }}
color: 0x00ff00
- name: Notify Discord (Failure)
if: failure() && steps.deploy_config.outputs.configured == 'true'
uses: sarisia/actions-status-discord@eb045afee445dc055c18d3d90bd0f244fd062708
with:
webhook: ${{ secrets.DISCORD_WEBHOOK }}
ack_no_webhook: true
title: "❌ Eliza Provisioning Worker Deploy Failed"
nodetail: true
description: |
Branch: develop
Commit: ${{ github.sha }}
color: 0xff0000
# ci-touch: re-trigger prod daemon deploy (dispatch runs were runner-starved) 2026-06-21
+200
View File
@@ -0,0 +1,200 @@
# Deploy packages/homepage to elizaOS/elizaos.github.io (org GitHub Pages site).
#
# What this does:
# 1. Builds packages/homepage (Vite static site) — the `prebuild` script in
# packages/homepage/package.json regenerates
# src/generated/release-data.ts from the GitHub Releases API before vite
# runs, so download buttons always reflect the latest published assets.
# 2. Writes a CNAME file (`eliza.app`) into dist/ so GitHub Pages serves the
# static output at https://eliza.app/. The org-pages URL
# https://elizaos.github.io/ also serves the same content (it redirects to
# eliza.app once the CNAME is in place).
# 3. Adds `.nojekyll` so GitHub Pages serves files/dirs that begin with `_`.
# 4. Pushes the built site to the `gh-pages` branch of
# elizaOS/elizaos.github.io using a deploy key. We deliberately push to
# `gh-pages` (not `main`) so the leaderboard source code on `main` stays
# untouched in git history.
#
# Required secret on this repo (elizaOS/eliza):
# HOMEPAGE_DEPLOY_KEY
# - SSH private key (PEM, e.g. `ssh-keygen -t ed25519 -C "homepage-deploy" -f homepage-deploy -N ""`).
# - The corresponding PUBLIC key must be registered as a Deploy Key with
# write access on the target repo elizaOS/elizaos.github.io.
#
# One-time setup on elizaOS/elizaos.github.io (the leaderboard repo):
# - In Settings → Pages, set source to:
# "Deploy from a branch" → Branch: `gh-pages` / `(root)`.
# This switches Pages off the leaderboard's old "GitHub Actions" source.
# - Disable or delete the leaderboard's existing `.github/workflows/deploy.yml`
# so it does not race with this workflow (its `actions/deploy-pages` calls
# become no-ops once Pages source is changed, but disabling it removes the
# spurious failed-run noise).
#
# DNS for eliza.app:
# - Add A records for the apex (eliza.app) pointing at GitHub Pages IPs:
# 185.199.108.153, 185.199.109.153, 185.199.110.153, 185.199.111.153
# - Or a CNAME record for www.eliza.app → elizaos.github.io.
# - In elizaos.github.io repo Settings → Pages, set Custom domain = eliza.app
# and enable "Enforce HTTPS" once the cert is provisioned.
name: Deploy Homepage
concurrency:
group: deploy-homepage
cancel-in-progress: true
on:
push:
branches: [develop]
paths:
- "packages/homepage/**"
- "packages/app-core/scripts/write-homepage-release-data.mjs"
- "packages/app-core/scripts/lib/asset-cdn.mjs"
- ".github/workflows/deploy-homepage.yml"
workflow_call:
inputs:
ref:
description: "Git ref to build the homepage from (defaults to the calling workflow ref)"
required: false
type: string
workflow_dispatch:
inputs:
ref:
description: "Git ref to build the homepage from"
required: false
type: string
permissions:
contents: write
jobs:
build-and-deploy:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 20
steps:
- name: Checkout source repo
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
ref: ${{ inputs.ref || github.ref }}
submodules: false
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: "canary"
- name: Install homepage dependencies
working-directory: packages/homepage
run: bun install --no-save --ignore-scripts
- name: Refresh homepage release data
working-directory: packages/homepage
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: node ../app-core/scripts/write-homepage-release-data.mjs
- name: Check homepage release data
id: release_data_check
working-directory: packages/homepage
shell: bash
run: |
if bun run check:release-data; then
echo "available=true" >> "$GITHUB_OUTPUT"
exit 0
fi
if [ "${{ github.event_name }}" = "workflow_call" ]; then
echo "::error::Release-triggered homepage deploys require published release data."
exit 1
fi
echo "::warning::No complete published release data is available; skipping homepage deploy for this develop push."
echo "available=false" >> "$GITHUB_OUTPUT"
- name: Build homepage
if: steps.release_data_check.outputs.available == 'true'
working-directory: packages/homepage
run: bun run build
- name: Install homepage browser
if: steps.release_data_check.outputs.available == 'true'
working-directory: packages/homepage
run: ./node_modules/.bin/playwright install --with-deps chromium
- name: Generate missing snapshot baselines
if: steps.release_data_check.outputs.available == 'true'
id: gen-baselines
working-directory: packages/homepage
run: |
SNAP_DIR="tests/e2e/visual.spec.ts-snapshots"
# 5 routes × 2 viewports × 1 browser project = 10 expected snapshots
EXPECTED=10
ACTUAL=$(ls "$SNAP_DIR"/*.png 2>/dev/null | wc -l || echo 0)
if [ "$ACTUAL" -lt "$EXPECTED" ]; then
echo "Found $ACTUAL of $EXPECTED snapshot baselines — regenerating all."
rm -f "$SNAP_DIR"/*.png 2>/dev/null || true
bun run test:e2e -- --update-snapshots || true
echo "generated=true" >> "$GITHUB_OUTPUT"
else
echo "generated=false" >> "$GITHUB_OUTPUT"
fi
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Commit new snapshot baselines
if: steps.release_data_check.outputs.available == 'true' && steps.gen-baselines.outputs.generated == 'true'
continue-on-error: true
run: |
SNAP_DIR="packages/homepage/tests/e2e/visual.spec.ts-snapshots"
if [ -n "$(git status --porcelain $SNAP_DIR)" ]; then
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add $SNAP_DIR
git commit -m "chore: generate visual regression snapshot baselines [skip ci]"
git push origin HEAD:${{ github.ref_name }} || (git pull --rebase origin ${{ github.ref_name }} && git push origin HEAD:${{ github.ref_name }})
echo "Pushed new snapshot baselines."
fi
- name: Test homepage downloads
if: steps.release_data_check.outputs.available == 'true' && steps.gen-baselines.outputs.generated == 'false'
working-directory: packages/homepage
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: bun run test:e2e
- name: Write CNAME and .nojekyll
if: steps.release_data_check.outputs.available == 'true'
run: |
echo "eliza.app" > packages/homepage/dist/CNAME
touch packages/homepage/dist/.nojekyll
- name: Check deploy key availability
if: steps.release_data_check.outputs.available == 'true'
id: deploy_key_check
shell: bash
env:
DEPLOY_KEY: ${{ secrets.HOMEPAGE_DEPLOY_KEY }}
run: |
if [ -n "$DEPLOY_KEY" ]; then
echo "available=true" >> "$GITHUB_OUTPUT"
else
echo "available=false" >> "$GITHUB_OUTPUT"
if [ "${{ github.event_name }}" = "workflow_call" ]; then
echo "::error::HOMEPAGE_DEPLOY_KEY secret not configured. Release-triggered homepage deploys must fail closed instead of silently skipping publication."
exit 1
fi
echo "::warning::HOMEPAGE_DEPLOY_KEY secret not configured. Skipping deploy step. The site was built successfully but will not be published. To enable publishing, register an SSH deploy key on elizaOS/elizaos.github.io and add the private key as HOMEPAGE_DEPLOY_KEY on this repo (see workflow header for details)."
fi
- name: Deploy to elizaOS/elizaos.github.io (gh-pages branch)
if: steps.release_data_check.outputs.available == 'true' && steps.deploy_key_check.outputs.available == 'true'
uses: peaceiris/actions-gh-pages@84c30a85c19949d7eee79c4ff27748b70285e453
with:
deploy_key: ${{ secrets.HOMEPAGE_DEPLOY_KEY }}
external_repository: elizaOS/elizaos.github.io
publish_branch: gh-pages
publish_dir: ./packages/homepage/dist
force_orphan: true
user_name: "github-actions[bot]"
user_email: "github-actions[bot]@users.noreply.github.com"
commit_message: "deploy: homepage from elizaOS/eliza@${{ github.sha }}"
+220
View File
@@ -0,0 +1,220 @@
name: Dev Smoke
on:
# Reusable entry for the scheduled exhaustive develop lane (#12342).
workflow_call:
pull_request:
branches: [main]
types: [opened, synchronize, reopened, ready_for_review, labeled]
push:
branches: [main, develop]
paths:
- ".github/workflows/dev-smoke.yml"
- "package.json"
- "bun.lock"
- "packages/app/**"
- "packages/app-core/**"
- "packages/core/**"
- "packages/shared/**"
- "packages/ui/**"
workflow_dispatch:
# CI fan-out fix: the PR-scoped concurrency from #14069 fell back to
# github.run_id on push, so every develop push got a unique group and
# nothing ever superseded. A 37-merge wave queued thousands of push runs
# and starved the self-hosted fleet. Fall back to github.ref (all develop
# pushes share a group) and cancel superseded runs on push too. PR behavior
# is unchanged (pull_request.number still wins the key); merge_group and
# schedule events are not in the cancel set, so the merge queue and nightly
# runs always complete.
concurrency:
group: dev-smoke-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }}
permissions:
contents: read
env:
CI: "true"
BUN_VERSION: "canary"
NODE_VERSION: "24.15.0"
NODE_NO_WARNINGS: "1"
# WHY: the dev smoke needs a real model when one is available, but forcing the
# lane through one provider makes develop red when that provider hits daily
# quota. The shared live-provider selector chooses from these keys in order and
# skips the Playwright spec cleanly when no usable provider is configured.
# Cerebras remains available to local/manual runs through the harness, but the
# scheduled CI lane does not force it after repeated daily-quota exhaustion.
# OpenAI is also intentionally omitted here: the live app currently sends a
# memory-response schema that OpenAI rejects strictly, which tests that
# provider contract rather than the dev/onboarding plumbing this lane owns.
CEREBRAS_API_KEY: ${{ secrets.CEREBRAS_API_KEY }}
GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_GENERATIVE_AI_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
ELIZAOS_CLOUD_API_KEY: ${{ secrets.ELIZAOS_CLOUD_API_KEY }}
jobs:
changes:
name: Classify changed paths
# Path classifier is a git-diff + node script: no self-hosted resources.
# Keep it on GitHub-hosted so a drained hetzner fleet cannot leave it
# queued indefinitely and pile up runs (#13617/#8501).
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 10
outputs:
dev_smoke: ${{ steps.filter.outputs.dev_smoke }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 0
submodules: false
- name: Setup Node.js for path gate
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Determine affected dev smoke surface
id: filter
shell: bash
env:
PR_LABELS: ${{ github.event_name == 'pull_request' && join(github.event.pull_request.labels.*.name, ',') || '' }}
run: |
set -euo pipefail
node packages/scripts/ci-path-gate.mjs \
--config dev-smoke \
--event "${{ github.event_name }}" \
--base "${{ github.event.pull_request.base.sha }}" \
--head "${{ github.event.pull_request.head.sha }}" \
--labels "$PR_LABELS" \
--output "$GITHUB_OUTPUT" \
--summary "$GITHUB_STEP_SUMMARY"
dev-smoke:
name: bun run dev onboarding chat
needs: changes
if: github.event_name != 'pull_request' || needs.changes.outputs.dev_smoke == 'true'
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 35
env:
PGLITE_WASM_MODE: node
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Free disk space for browser smoke
run: |
set -euxo pipefail
df -h /
sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
docker system prune -af || true
df -h /
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Install Playwright Chromium
run: bunx playwright install --with-deps chromium
- name: Generate shared i18n assets
run: bun run --cwd packages/shared build:i18n
# Without a prebuilt dist/, the renderer's vite dev server serves
# @elizaos/core from its source graph (400+ on-demand module transforms),
# which is too slow on CI for the chat composer to mount inside the test's
# 90s window. Prebuilding core (and its workspace deps) makes the dev server
# serve the compiled browser bundle instead. See resolveElizaCoreBundlePath
# in packages/app/vite.config.ts.
# @elizaos/shared is also built because its `./*` subpath exports map to a
# gitignored dist/ (e.g. @elizaos/shared/utils/permission-deep-links, used
# by PermissionRecoveryCallout). Without it the dev server can't resolve
# the subpath and the smoke fails at import time.
- name: Build @elizaos/core browser bundle
run: bunx turbo run build --filter=@elizaos/core --filter=@elizaos/shared
- name: Run bun dev onboarding chat smoke
run: bun run --cwd packages/app test:dev-smoke
# The smoke has historically failed inside gotoChatComposer (the /chat
# composer never paints) with no captured renderer state, making the
# actual blocking gate impossible to pin down. Upload the Playwright
# trace/screenshot/error-context so the failure is diagnosable.
- name: Upload dev-smoke artifacts
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: dev-smoke-results
path: |
packages/app/playwright-report/
packages/app/test-results/
if-no-files-found: ignore
hmr:
name: Vite HMR dependency-level smoke
needs: changes
if: github.event_name != 'pull_request' || needs.changes.outputs.dev_smoke == 'true'
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 35
env:
PGLITE_WASM_MODE: node
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Free disk space for browser smoke
run: |
set -euxo pipefail
df -h /
sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
docker system prune -af || true
df -h /
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Install Playwright Chromium
run: bunx playwright install --with-deps chromium
- name: Generate shared i18n assets
run: bun run --cwd packages/shared build:i18n
- name: Build @elizaos/core browser bundle
run: bunx turbo run build --filter=@elizaos/core --filter=@elizaos/shared
- name: Run Vite HMR dependency-level smoke
run: bun run test:hmr
- name: Upload HMR artifacts
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: hmr-results
path: |
packages/app/playwright-report/
packages/app/test-results/hmr/
if-no-files-found: ignore
+158
View File
@@ -0,0 +1,158 @@
name: Develop Exhaustive Lane
# Un-cancellable scheduled run of the FULL develop matrix (#12342, epic #12191
# phase 5). test.yml's own 09:17 schedule already covers the unit / e2e /
# plugin / desktop / zero-key lanes; this orchestrator adds the platform lanes
# that otherwise only run path-gated on PRs — Windows, mobile, scenario, the
# three UI gates, the keyless harness, docker, dev onboarding, and the
# electrobun/desktop contract — by invoking each via `workflow_call` so no step
# is duplicated. workflow_call bypasses the per-workflow path filters, and a
# schedule-origin event makes every called workflow's
# `github.event_name != 'pull_request'` job gate evaluate true, so the lanes run
# unfiltered (exhaustive). The final proof job statically asserts the committed
# manifest still matches test.yml AND that every required reusable workflow is
# wired here and declares workflow_call, then fails the run if any reusable lane
# did not succeed — a skipped or failed platform lane is a coverage gap, not a
# pass.
on:
schedule:
# Twice daily, offset from test.yml (09:17) and ci-full-matrix-proof (03:13/
# 15:13) so the three scheduled signals do not stampede the runner pool.
- cron: "0 6 * * *"
- cron: "0 18 * * *"
workflow_dispatch:
# Dedicated group, never cancelled by push/PR traffic: the whole point is a
# completed daily coverage signal (#12337/#12342).
concurrency:
group: develop-exhaustive-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: read
packages: read
actions: read
jobs:
windows:
name: Windows full matrix
uses: ./.github/workflows/windows-ci.yml
secrets: inherit
mobile:
name: Mobile build smoke
uses: ./.github/workflows/mobile-build-smoke.yml
secrets: inherit
scenario:
name: Scenario deterministic E2E
uses: ./.github/workflows/scenario-pr.yml
secrets: inherit
ui-e2e:
name: UI e2e gate
uses: ./.github/workflows/ui-e2e-gate.yml
secrets: inherit
ui-fixture:
name: UI fixture e2e
uses: ./.github/workflows/ui-fixture-e2e.yml
secrets: inherit
ui-story:
name: UI story gate
uses: ./.github/workflows/ui-story-gate.yml
secrets: inherit
keyless-harness:
name: Keyless harness E2E
uses: ./.github/workflows/keyless-harness-e2e.yml
secrets: inherit
docker:
name: Docker image smoke
uses: ./.github/workflows/docker-ci-smoke.yml
secrets: inherit
dev-onboarding:
name: Dev onboarding smoke
uses: ./.github/workflows/dev-smoke.yml
secrets: inherit
electrobun:
name: Electrobun desktop release
uses: ./.github/workflows/test-electrobun-release.yml
secrets: inherit
matrix-proof:
name: Exhaustive lane matrix proof
needs:
- windows
- mobile
- scenario
- ui-e2e
- ui-fixture
- ui-story
- keyless-harness
- docker
- dev-onboarding
- electrobun
# Run even when a reusable lane failed so the proof enumerates every lane
# and the result check below can name the offender; only a cancel skips it.
if: ${{ !cancelled() }}
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
env:
CI: "true"
BUN_VERSION: "1.3.14"
NODE_VERSION: "24.15.0"
NODE_NO_WARNINGS: "1"
NODE_OPTIONS: "--max-old-space-size=8192"
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Assert every reusable lane succeeded (skipped/failed == gap)
env:
RESULTS: >-
windows=${{ needs.windows.result }}
mobile=${{ needs.mobile.result }}
scenario=${{ needs.scenario.result }}
ui-e2e=${{ needs.ui-e2e.result }}
ui-fixture=${{ needs.ui-fixture.result }}
ui-story=${{ needs.ui-story.result }}
keyless-harness=${{ needs.keyless-harness.result }}
docker=${{ needs.docker.result }}
dev-onboarding=${{ needs.dev-onboarding.result }}
electrobun=${{ needs.electrobun.result }}
run: |
fail=0
for pair in $RESULTS; do
lane="${pair%%=*}"
result="${pair##*=}"
if [ "$result" = "success" ]; then
echo "lane $lane: success"
else
echo "::error::exhaustive lane $lane did not succeed (result=$result) — a skipped or failed platform lane is a coverage gap, not a pass"
fail=1
fi
done
exit "$fail"
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: ${{ env.BUN_VERSION }}
install-command: bun install --frozen-lockfile --ignore-scripts
install-native-deps: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Validate GitHub-native turbo cache contract
run: node packages/scripts/ci-turbo-cache-contract.mjs
- name: Validate workflow dedup contract
run: node packages/scripts/ci-workflow-dedup-contract.mjs
- name: Capture exhaustive test plan
run: node packages/scripts/run-all-tests.mjs --plan=json > /tmp/full-matrix-plan.json
- name: Prove exhaustive lane coverage (manifest + reusable wiring)
run: node packages/scripts/ci-full-matrix-proof.mjs --plan-file /tmp/full-matrix-plan.json
+146
View File
@@ -0,0 +1,146 @@
# Nightly post-merge live sweep — the only place TEST_LANE=post-merge runs the
# whole workspace matrix. The PR/develop gates are deliberately deterministic
# and secret-free, which means every *.real.test.ts / *.live.test.ts suite is
# excluded there; without this lane, live-API coverage exists only as a plan.
# run-all-tests' post-merge lane prints the guarded-suite accounting (armed /
# missing-creds / opt-in / blocked, from packages/scripts/lib/real-live-suites.mjs)
# and hard-fails on manifest drift, so a missing secret is a loud named skip,
# never a silent green.
#
# Scheduled runs are gated on vars.ELIZA_DEVELOP_LIVE_ENABLED == 'true' so the
# lane can be armed org-side once runner capacity allows (#13309); manual
# dispatch always works. Local operators get the same sweep interactively via
# `bun run test:console` (packages/scripts/test-console).
name: Develop Live Sweep
on:
workflow_dispatch:
inputs:
filter:
description: "Optional --filter regex over task labels (default: whole matrix)"
type: string
default: ""
schedule:
# 04:30 UTC — clear of develop-exhaustive (06:00/18:00) and test.yml (09:17).
- cron: "30 4 * * *"
permissions:
contents: read
concurrency:
group: develop-live
cancel-in-progress: false
jobs:
live-sweep:
name: Post-merge live matrix
if: github.event_name == 'workflow_dispatch' || vars.ELIZA_DEVELOP_LIVE_ENABLED == 'true'
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 480
env:
TEST_LANE: post-merge
ELIZA_LIVE_TEST: "1"
ELIZA_REAL_APIS: "1"
ELIZA_SKIP_ARTIFACT_SYNC: "1"
# Canonical post-merge secret set (packages/scripts/post-merge-secrets.txt).
# A missing secret self-skips its suites with a named accounting line.
CEREBRAS_API_KEY: ${{ secrets.CEREBRAS_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_GENERATIVE_AI_API_KEY }}
OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
XAI_API_KEY: ${{ secrets.XAI_API_KEY }}
ELEVENLABS_API_KEY: ${{ secrets.ELEVENLABS_API_KEY }}
DISCORD_BOT_TOKEN: ${{ secrets.DISCORD_BOT_TOKEN }}
DISCORD_TEST_GUILD_ID: ${{ secrets.DISCORD_TEST_GUILD_ID }}
SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
SLACK_TEST_CHANNEL_ID: ${{ secrets.SLACK_TEST_CHANNEL_ID }}
TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }}
TELEGRAM_TEST_CHAT_ID: ${{ secrets.TELEGRAM_TEST_CHAT_ID }}
WHATSAPP_TOKEN: ${{ secrets.WHATSAPP_TOKEN }}
WHATSAPP_PHONE_NUMBER_ID: ${{ secrets.WHATSAPP_PHONE_NUMBER_ID }}
X_BEARER_TOKEN: ${{ secrets.X_BEARER_TOKEN }}
# The ephemeral Actions token satisfies the read-only GitHub live suites.
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
LINEAR_API_KEY: ${{ secrets.LINEAR_API_KEY }}
CALENDLY_API_KEY: ${{ secrets.CALENDLY_API_KEY }}
CALENDLY_ACCESS_TOKEN: ${{ secrets.CALENDLY_ACCESS_TOKEN }}
TWILIO_ACCOUNT_SID: ${{ secrets.TWILIO_ACCOUNT_SID }}
TWILIO_AUTH_TOKEN: ${{ secrets.TWILIO_AUTH_TOKEN }}
GOOGLE_OAUTH_REFRESH_TOKEN: ${{ secrets.GOOGLE_OAUTH_REFRESH_TOKEN }}
BLUESKY_PASSWORD: ${{ secrets.BLUESKY_PASSWORD }}
FARCASTER_NEYNAR_API_KEY: ${{ secrets.FARCASTER_NEYNAR_API_KEY }}
SHOPIFY_API_KEY: ${{ secrets.SHOPIFY_API_KEY }}
HYPERLIQUID_PRIVATE_KEY: ${{ secrets.HYPERLIQUID_PRIVATE_KEY }}
POLYMARKET_API_KEY: ${{ secrets.POLYMARKET_API_KEY }}
STRAVA_ACCESS_TOKEN: ${{ secrets.STRAVA_ACCESS_TOKEN }}
OURA_ACCESS_TOKEN: ${{ secrets.OURA_ACCESS_TOKEN }}
FITBIT_ACCESS_TOKEN: ${{ secrets.FITBIT_ACCESS_TOKEN }}
WITHINGS_ACCESS_TOKEN: ${{ secrets.WITHINGS_ACCESS_TOKEN }}
GOOGLE_CALENDAR_ACCESS_TOKEN: ${{ secrets.GOOGLE_CALENDAR_ACCESS_TOKEN }}
ATLASCLOUD_API_KEY: ${{ secrets.ATLASCLOUD_API_KEY }}
TAVILY_API_KEY: ${{ secrets.TAVILY_API_KEY }}
BIRDEYE_API_KEY: ${{ secrets.BIRDEYE_API_KEY }}
ELIZAOS_CLOUD_API_KEY: ${{ secrets.ELIZAOS_CLOUD_API_KEY }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24.15.0"
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: "1.3.14" # pinned: floating canary writes lockfileVersion 2 and breaks --frozen-lockfile (#11184/#9454)
install-command: bun install
- name: Build workspace
run: bun run build
# plugin-sql's RLS *.real.test.ts suites gate on POSTGRES_URL with the
# same URL run-all-tests auto-provisions locally. Best-effort: if the
# runner has no docker, the suites self-skip with a named accounting line.
- name: Provision Postgres (best-effort)
run: |
if command -v docker >/dev/null 2>&1; then
docker rm -f develop-live-postgres 2>/dev/null || true
docker run -d --name develop-live-postgres \
-e POSTGRES_USER=eliza_test -e POSTGRES_PASSWORD=test123 -e POSTGRES_DB=eliza_test \
-p 127.0.0.1:5433:5432 postgres:17
echo "POSTGRES_URL=postgresql://eliza_test:test123@127.0.0.1:5433/eliza_test" >> "$GITHUB_ENV"
else
echo "docker unavailable; plugin-sql Postgres real suites will self-skip"
fi
- name: Post-merge live matrix
env:
RUNNER_TEMP_LOG: ${{ runner.temp }}/develop-live-sweep.log
TEST_FILTER: ${{ inputs.filter }}
run: |
set -o pipefail
filter_args=()
if [ -n "${TEST_FILTER:-}" ]; then
filter_args+=(--filter="$TEST_FILTER")
fi
node packages/scripts/run-all-tests.mjs --all --no-cloud "${filter_args[@]}" 2>&1 | tee "$RUNNER_TEMP_LOG"
- name: Cloud unit suite
if: always()
run: bun run test:cloud
- name: Teardown Postgres
if: always()
run: docker rm -f develop-live-postgres 2>/dev/null || true
- name: Upload sweep log
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: develop-live-sweep-log
path: ${{ runner.temp }}/develop-live-sweep.log
if-no-files-found: ignore
+178
View File
@@ -0,0 +1,178 @@
name: Develop PR
# Lightweight gate for pull requests targeting `develop`: lint, typecheck, and
# build only. The full test/e2e/scenario/benchmark surface runs post-merge on
# push to `develop` (and nightly/on-demand), not on every PR — so PR feedback
# stays fast and hosted-runner capacity is spent on the ~200 daily PRs' fast
# signal rather than the whole integration matrix. `main` PRs keep the heavy
# gates via `ci.yaml`, `test.yml`'s peers, and `quality.yml`. Steps mirror
# `ci.yaml` so PR and post-merge lanes stay in agreement.
on:
pull_request:
branches: [develop]
types: [opened, synchronize, reopened, ready_for_review]
concurrency:
group: develop-pr-${{ github.event.pull_request.number }}
cancel-in-progress: true
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
lane-coverage:
name: Test Integrity (lane coverage)
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
# Full history: the error-policy ratchet diffs every touched file
# against its content at the merge-base with origin/develop.
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24.15.0"
- name: Per-plugin e2e coverage
run: node packages/scripts/lint-lane-coverage.mjs
# Binding policy audits from CLAUDE.md that previously ran nowhere in
# CI. Both are dependency-free node scripts (<5s): the error-policy
# ratchet fails only when a file this PR touches ADDS an empty catch or
# server-side console.* call; test-integrity statically lints test
# sources for larp patterns.
- name: Error-policy ratchet (diff-scoped)
run: |
git fetch --no-tags origin develop:refs/remotes/origin/develop || true
node packages/scripts/error-policy-ratchet.mjs
- name: Test-integrity audit
run: node packages/scripts/lint-test-integrity.mjs
- name: Test-integrity audit self-test
run: node packages/scripts/lint-test-integrity.self-test.mjs
- name: Changed-file coverage classifier self-tests
run: |
while IFS= read -r self_test; do
if [ ! -f "$self_test" ]; then
echo "registered coverage self-test is missing: $self_test"
exit 1
fi
node "$self_test"
done < scripts/security/coverage-node-self-tests.txt
# View→action ratchet (#14369): every builtin-view on-screen mutation
# must map to a registered agent action (chat twin), and the generated
# action catalog must reflect the real registered surface. Dependency-free
# (node stdlib only) like the other audits in this job.
- name: View-action ratchet self-test
run: node packages/scripts/view-action-ratchet.mjs --self-test
- name: View-action ratchet
run: node packages/scripts/view-action-ratchet.mjs
lint:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24.15.0"
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: "1.3.14" # pinned: floating canary writes lockfileVersion 2 and breaks --frozen-lockfile (#11184/#9454)
install-command: bun install
install-native-deps: "false"
install-protoc: "false"
setup-python: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Run lint
run: bun run lint
- name: Check formatting
run: bun run format:check
typecheck:
runs-on: ubuntu-latest
# 35 not 20: at --concurrency=4 a near-total affected cone (~250 packages)
# needs headroom — the one green full-cone run spent 11 minutes in the
# typecheck step alone, on top of checkout/install/build:core (#15140).
timeout-minutes: 35
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
# Full history so `turbo run typecheck --affected` can resolve the PR
# merge base (#12341); a shallow clone degrades to typechecking all.
fetch-depth: 0
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24.15.0"
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: "1.3.14" # pinned: floating canary writes lockfileVersion 2 and breaks --frozen-lockfile (#11184/#9454)
install-command: bun install
install-native-deps: "false"
install-protoc: "false"
setup-python: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
# Typecheck resolves `@elizaos/core` types from its built dist, so build
# core before running tsc across the affected cone.
- name: Build core
run: bun run build:core
# --concurrency=4 not 8: hosted ubuntu-latest runners have 4 vCPUs and
# 16 GB RAM, and when the affected cone goes near-total (a base-branch
# commit touching turbo globalDependencies like root package.json /
# turbo.json makes every stale-base PR typecheck all ~250 packages) eight
# concurrent tsgo processes exhaust the runner — the OS SIGKILLs tasks or
# the runner VM itself dies (#15140). NODE_OPTIONS only bounds node-based
# tools, not native tsgo, so concurrency is the memory lever here.
- name: Run typecheck (affected)
run: NODE_OPTIONS='--max-old-space-size=8192' node packages/scripts/run-turbo.mjs run typecheck --concurrency=4 --affected
env:
TURBO_SCM_BASE: ${{ github.event.pull_request.base.sha }}
build:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24.15.0"
- name: Setup workspace dependencies
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: "1.3.14" # pinned: floating canary writes lockfileVersion 2 and breaks --frozen-lockfile (#11184/#9454)
install-command: bun install
install-native-deps: "false"
install-protoc: "false"
setup-python: "false"
skip-avatar-clone: "true"
no-vision-deps: "true"
- name: Build packages
run: bun run build:core
+165
View File
@@ -0,0 +1,165 @@
name: Develop Staging Beta Release
on:
workflow_run:
workflows: ["Cloud CF Deploy"]
types: [completed]
workflow_dispatch:
inputs:
target_sha:
description: Develop commit SHA to release. Defaults to the workflow ref SHA.
required: false
type: string
permissions:
actions: write
checks: read
contents: read
concurrency:
group: develop-staging-beta-${{ github.event.workflow_run.head_sha || inputs.target_sha || github.sha }}
cancel-in-progress: false
jobs:
gate-and-release:
name: Gate Develop and Dispatch Beta
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
if: >-
${{
github.event_name == 'workflow_dispatch' ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.head_branch == 'develop' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_repository.full_name == github.repository
)
}}
steps:
- name: Resolve target commit
id: target
env:
EVENT_NAME: ${{ github.event_name }}
WORKFLOW_RUN_SHA: ${{ github.event.workflow_run.head_sha || '' }}
INPUT_SHA: ${{ inputs.target_sha || '' }}
run: |
if [ "$EVENT_NAME" = "workflow_run" ]; then
TARGET_SHA="$WORKFLOW_RUN_SHA"
elif [ -n "$INPUT_SHA" ]; then
TARGET_SHA="$INPUT_SHA"
else
TARGET_SHA="${GITHUB_SHA}"
fi
if [ -z "$TARGET_SHA" ]; then
echo "::error::Unable to resolve target SHA."
exit 1
fi
echo "sha=$TARGET_SHA" >> "$GITHUB_OUTPUT"
echo "Develop staging gate target: $TARGET_SHA" >> "$GITHUB_STEP_SUMMARY"
- name: Ensure target is current develop
env:
GH_TOKEN: ${{ github.token }}
TARGET_SHA: ${{ steps.target.outputs.sha }}
run: |
DEVELOP_SHA="$(gh api "repos/${GITHUB_REPOSITORY}/git/ref/heads/develop" --jq '.object.sha')"
echo "develop HEAD: ${DEVELOP_SHA}" >> "$GITHUB_STEP_SUMMARY"
if [ "$TARGET_SHA" != "$DEVELOP_SHA" ]; then
echo "Target ${TARGET_SHA} is no longer develop HEAD (${DEVELOP_SHA}); a newer develop run will handle staging beta." >> "$GITHUB_STEP_SUMMARY"
echo "superseded=true" >> "$GITHUB_ENV"
fi
- name: Wait for all checks on target commit
if: env.superseded != 'true'
env:
GH_TOKEN: ${{ github.token }}
TARGET_SHA: ${{ steps.target.outputs.sha }}
TIMEOUT_SECONDS: ${{ vars.DEVELOP_STAGING_GATE_TIMEOUT_SECONDS || '3600' }}
run: |
set -euo pipefail
deadline=$((SECONDS + TIMEOUT_SECONDS))
ignored_names_json='["Develop Staging Beta Release","NPM Release","Release Orchestrator"]'
while [ "$SECONDS" -lt "$deadline" ]; do
DEVELOP_SHA="$(gh api "repos/${GITHUB_REPOSITORY}/git/ref/heads/develop" --jq '.object.sha')"
if [ "$TARGET_SHA" != "$DEVELOP_SHA" ]; then
echo "Target ${TARGET_SHA} is no longer develop HEAD (${DEVELOP_SHA}); a newer develop run will handle staging beta." >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
gh api \
"repos/${GITHUB_REPOSITORY}/actions/runs?head_sha=${TARGET_SHA}&per_page=100" \
> /tmp/workflow-runs.json
set +e
IGNORED_NAMES_JSON="$ignored_names_json" node <<'NODE'
const fs = require("node:fs");
const payload = JSON.parse(fs.readFileSync("/tmp/workflow-runs.json", "utf8"));
const runs = payload.workflow_runs ?? [];
const ignoredNames = new Set(JSON.parse(process.env.IGNORED_NAMES_JSON));
const currentRunId = Number(process.env.GITHUB_RUN_ID);
const successfulConclusions = new Set(["success", "neutral", "skipped"]);
const relevant = runs.filter((run) => {
if (Number(run.id) === currentRunId) return false;
if (ignoredNames.has(run.name)) return false;
return run.head_sha === process.env.TARGET_SHA;
});
const failed = relevant.filter(
(run) => run.status === "completed" && !successfulConclusions.has(run.conclusion),
);
const pending = relevant.filter((run) => run.status !== "completed");
console.log(`Observed ${relevant.length} workflow run(s) for ${process.env.TARGET_SHA}.`);
if (pending.length > 0) {
console.log(`Pending: ${pending.map((run) => `${run.name}/${run.status}`).join(", ")}`);
}
if (failed.length > 0) {
console.error(
`Failed: ${failed.map((run) => `${run.name}/${run.conclusion}`).join(", ")}`,
);
process.exit(2);
}
if (pending.length > 0) process.exit(1);
if (relevant.length === 0) {
console.log("No completed workflow runs are visible yet.");
process.exit(1);
}
NODE
status=$?
set -e
if [ "$status" -eq 0 ]; then
echo "All observed workflows are green for ${TARGET_SHA}." >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
if [ "$status" -eq 2 ]; then
DEVELOP_SHA="$(gh api "repos/${GITHUB_REPOSITORY}/git/ref/heads/develop" --jq '.object.sha')"
if [ "$TARGET_SHA" != "$DEVELOP_SHA" ]; then
echo "Target ${TARGET_SHA} was superseded by ${DEVELOP_SHA}; ignoring cancelled checks from the old head." >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
echo "::error::At least one workflow failed for ${TARGET_SHA}; beta release blocked."
exit 1
fi
sleep 30
done
echo "::error::Timed out waiting for all workflows to finish for ${TARGET_SHA}."
exit 1
- name: Dispatch beta release
if: env.superseded != 'true'
env:
GH_TOKEN: ${{ github.token }}
AUTO_BETA_RELEASE_FROM_DEVELOP: ${{ vars.AUTO_BETA_RELEASE_FROM_DEVELOP || 'true' }}
TARGET_SHA: ${{ steps.target.outputs.sha }}
run: |
if [ "$AUTO_BETA_RELEASE_FROM_DEVELOP" != "true" ]; then
echo "AUTO_BETA_RELEASE_FROM_DEVELOP is not true; beta release dispatch skipped." >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
gh workflow run release.yaml --repo "$GITHUB_REPOSITORY" --ref develop -f release_type=beta
echo "Dispatched release.yaml beta workflow for develop after ${TARGET_SHA} passed staging." >> "$GITHUB_STEP_SUMMARY"
+203
View File
@@ -0,0 +1,203 @@
name: Docker CI Smoke
on:
# Reusable entry for the scheduled exhaustive develop lane (#12342).
workflow_call:
push:
branches: [develop]
schedule:
# Nightly heavy lane (#14051): 07:00 UTC.
- cron: "0 7 * * *"
workflow_dispatch:
# CI fan-out fix: the PR-scoped concurrency from #14069 fell back to
# github.run_id on push, so every develop push got a unique group and
# nothing ever superseded. A 37-merge wave queued thousands of push runs
# and starved the self-hosted fleet. Fall back to github.ref (all develop
# pushes share a group) and cancel superseded runs on push too. PR behavior
# is unchanged (pull_request.number still wins the key); merge_group and
# schedule events are not in the cancel set, so the merge queue and nightly
# runs always complete.
concurrency:
group: docker-ci-smoke-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }}
permissions:
contents: read
jobs:
changes:
name: Classify changed paths
if: github.event_name != 'pull_request' || contains(github.event.pull_request.labels.*.name, 'ci:full')
# Path classifier is a git-diff + node script: no self-hosted resources.
# Keep it on GitHub-hosted so a drained hetzner fleet cannot leave it
# queued indefinitely and pile up runs (#13617/#8501).
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 10
outputs:
docker: ${{ steps.filter.outputs.docker }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 0
submodules: false
- name: Setup Node.js for path gate
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24"
- name: Determine affected Docker smoke surface
id: filter
shell: bash
env:
PR_LABELS: ${{ github.event_name == 'pull_request' && join(github.event.pull_request.labels.*.name, ',') || '' }}
run: |
set -euo pipefail
node packages/scripts/ci-path-gate.mjs \
--config docker \
--event "${{ github.event_name }}" \
--base "${{ github.event.pull_request.base.sha }}" \
--head "${{ github.event.pull_request.head.sha }}" \
--labels "$PR_LABELS" \
--output "$GITHUB_OUTPUT" \
--summary "$GITHUB_STEP_SUMMARY"
docker-ci-smoke:
name: Build production Docker image (+ smoke boot)
needs: changes
if: github.event_name != 'pull_request' || needs.changes.outputs.docker == 'true'
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 90
env:
BUN_VERSION: "canary"
DOCKER_IMAGE: elizaos/agent:docker-smoke
SMOKE_TIMEOUT_SEC: "420"
SMOKE_PORT: "32138"
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: "24"
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Cache Bun install
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9
with:
path: ~/.bun/install/cache
key: bun-${{ runner.os }}-${{ hashFiles('bun.lock') }}
restore-keys: |
bun-${{ runner.os }}-
- name: Free disk space for Docker smoke
run: |
df -h
# github-hosted-only disk cleanup; self-hosted runners (#8501) have no
# passwordless sudo and don't need it, so never fail the job on it.
sudo rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc /opt/hostedtoolcache/CodeQL || true
docker system prune -af --volumes || true
df -h
- name: Run Docker CI smoke path
timeout-minutes: 45
# Run the smoke script in its own process group so a wedged docker
# child cannot outlive the wrapper timeout and keep the step active.
run: |
set -euo pipefail
stop_smoke() {
if [[ -n "${smoke_pid:-}" ]] && kill -0 "$smoke_pid" 2>/dev/null; then
kill -TERM "-$smoke_pid" 2>/dev/null || kill -TERM "$smoke_pid" 2>/dev/null || true
sleep 10
kill -KILL "-$smoke_pid" 2>/dev/null || kill -KILL "$smoke_pid" 2>/dev/null || true
fi
}
trap 'stop_smoke; exit 130' INT
trap 'stop_smoke; exit 143' TERM
mkdir -p .tmp/qa
smoke_log=".tmp/qa/docker-ci-smoke-step.log"
setsid bash packages/app-core/scripts/docker-ci-smoke.sh --tag gha-${{ github.run_id }} > >(tee "$smoke_log") 2> >(tee -a "$smoke_log" >&2) &
smoke_pid=$!
smoke_started_at=$SECONDS
last_heartbeat=$SECONDS
deadline=$((SECONDS + 40 * 60))
while kill -0 "$smoke_pid" 2>/dev/null; do
if (( SECONDS >= deadline )); then
echo "::error::Docker CI smoke path exceeded 40 minutes"
kill -TERM "-$smoke_pid" 2>/dev/null || kill -TERM "$smoke_pid" 2>/dev/null || true
sleep 15
kill -KILL "-$smoke_pid" 2>/dev/null || kill -KILL "$smoke_pid" 2>/dev/null || true
wait "$smoke_pid" || true
exit 124
fi
if (( SECONDS - last_heartbeat >= 60 )); then
last_heartbeat=$SECONDS
echo "[docker-ci-smoke] still running after $((SECONDS - smoke_started_at))s; log: $smoke_log"
fi
sleep 10
done
set +e
wait "$smoke_pid"
smoke_status=$?
set -e
if (( smoke_status != 0 )); then
echo "::group::Docker CI smoke step tail"
tail -n 240 "$smoke_log" || true
echo "::endgroup::"
echo "::error::Docker CI smoke failed with exit code ${smoke_status}; see the step tail and uploaded .tmp/qa artifacts."
exit "$smoke_status"
fi
# Inline log capture runs on failure (not cancelled).
- name: Dump container log + inspect (inline)
if: failure()
timeout-minutes: 2
run: |
echo "::group::Docker containers"
timeout 15 docker ps -a || echo "(docker ps timed out)"
echo "::endgroup::"
timeout 15 docker ps -a --format '{{.Names}}' > /tmp/eliza-docker-containers.txt || true
for C in $(grep '^eliza-docker-smoke-' /tmp/eliza-docker-containers.txt || true); do
echo "::group::docker logs $C"
timeout 30 docker logs "$C" 2>&1 || echo "(docker logs timed out)"
echo "::endgroup::"
echo "::group::docker inspect $C (state)"
timeout 15 docker inspect "$C" --format '{{json .State}}' 2>&1 | python3 -m json.tool || true
echo "::endgroup::"
done
- name: Upload container logs + inspect
if: failure()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: docker-ci-smoke-artifacts-${{ github.run_id }}
path: |
.tmp/qa/docker-ci-smoke-*
/tmp/eliza-docker-*.txt
if-no-files-found: warn
retention-days: 7
- name: Summarize Docker smoke result
if: always()
run: |
echo "## Docker CI Smoke" >> "$GITHUB_STEP_SUMMARY"
echo "" >> "$GITHUB_STEP_SUMMARY"
echo "- Production Dockerfile: \`packages/app-core/deploy/Dockerfile.ci\`" >> "$GITHUB_STEP_SUMMARY"
echo "- Script: \`packages/app-core/scripts/docker-ci-smoke.sh\`" >> "$GITHUB_STEP_SUMMARY"
echo "- Trigger: \`${{ github.event_name }}\`" >> "$GITHUB_STEP_SUMMARY"
echo "- Ref: \`${{ github.ref }}\`" >> "$GITHUB_STEP_SUMMARY"
echo "- Smoke endpoints: \`/api/health\` then fallback \`/api/status\`" >> "$GITHUB_STEP_SUMMARY"
+239
View File
@@ -0,0 +1,239 @@
name: Docs CI
# Workflow for documentation quality checks
# Combines dead link checking and quality improvements
concurrency:
group: docs-ci-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
on:
pull_request:
types: [opened, ready_for_review]
paths:
- "packages/docs/**/*.mdx"
- "packages/docs/**/*.md"
- "packages/docs/docs.json"
- "packages/docs/mint.json"
# Allow manual triggering
workflow_dispatch:
inputs:
check_type:
description: "Type of check to run"
required: true
type: choice
options:
- all
- links
- quality
permissions:
contents: write
pull-requests: write
issues: write
id-token: write
jobs:
# ===========================================
# Check Dead Links
# ===========================================
check-links:
if: |
github.event.pull_request.draft != true &&
(github.event.inputs.check_type == 'all' || github.event.inputs.check_type == 'links' || github.event.inputs.check_type == '')
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- name: Checkout repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 0
filter: blob:none
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Git
run: |
git config --global user.name "github-actions[bot]"
git config --global user.email "github-actions[bot]@users.noreply.github.com"
- name: Check and Fix Dead Links with Claude
id: claude-fix-links
# Don't fail docs CI when the Anthropic API is unavailable (low
# credit balance, rate limit, transient outage). The fix pass is
# best-effort; downstream "Check for Changes" steps already handle
# the no-op path.
continue-on-error: true
uses: anthropics/claude-code-action@ba0aafd4308cbba7165f9f2cdb0cfbed5a3c99ce
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
prompt: |
Check all links in the packages/docs/ directory for broken or dead links.
**Instructions:**
1. Scan all MDX and MD files in packages/docs/
2. Check for internal relative links, external links, and anchor links
3. For each broken link found:
- Typos: Correct the spelling/path
- Moved files: Update to new location
- Missing anchors: Remove or update to valid anchor
- External 404s: Comment out with explanation
**Link Checking Rules:**
- Internal links should point to existing files in packages/docs/
- Skip checking localhost and example.com links
- Use fuzzy matching (85% threshold) for path corrections
**Output:** Report findings but only make high-confidence fixes.
claude_args: |
--model claude-sonnet-4-6
--allowedTools "Bash(find packages/docs -name '*.mdx' -o -name '*.md'),Bash(grep -n),Bash(curl -s -o /dev/null -w '%{http_code}'),Read,EditFile(packages/docs/*)"
- name: Check for Changes
id: check-link-changes
run: |
if [ -n "$(git status --porcelain packages/docs/)" ]; then
echo "changes=true" >> $GITHUB_OUTPUT
else
echo "changes=false" >> $GITHUB_OUTPUT
fi
- name: Commit Link Fixes
if: steps.check-link-changes.outputs.changes == 'true'
run: |
git add packages/docs/
git commit -m "fix(docs): automated broken link fixes [skip ci]" || true
# ===========================================
# Check Documentation Quality
# ===========================================
check-quality:
if: |
github.event.pull_request.draft != true &&
(github.event.inputs.check_type == 'all' || github.event.inputs.check_type == 'quality' || github.event.inputs.check_type == '')
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- name: Checkout repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 0
filter: blob:none
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Git
run: |
git config --global user.name "github-actions[bot]"
git config --global user.email "github-actions[bot]@users.noreply.github.com"
- name: Check and Fix Quality Issues with Claude
id: claude-fix-quality
# Don't fail docs CI when the Anthropic API is unavailable (low
# credit balance, rate limit, transient outage). The fix pass is
# best-effort; downstream "Check for Changes" steps already handle
# the no-op path.
continue-on-error: true
uses: anthropics/claude-code-action@ba0aafd4308cbba7165f9f2cdb0cfbed5a3c99ce
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
prompt: |
Check all MDX files in packages/docs/ for quality issues.
**Quality Checks:**
1. Double Header Issues - MDX files with frontmatter title AND H1 heading
2. Missing frontmatter fields (title or description)
3. Inconsistent heading hierarchy (H3 following H1 without H2)
4. Code blocks without language tags
**For Double Headers:**
- Remove the H1 heading from content (keep only frontmatter title)
- Mintlify auto-generates H1 from frontmatter
**Fix Application Rules:**
- Make minimal, precise changes
- Preserve all content except what needs fixing
- Maintain proper MDX formatting
**Output:** Report findings and apply fixes.
claude_args: |
--model claude-sonnet-4-6
--allowedTools "Bash(find packages/docs -name '*.mdx'),Bash(grep -n),Read,EditFile(packages/docs/*)"
- name: Check for Changes
id: check-quality-changes
run: |
if [ -n "$(git status --porcelain packages/docs/)" ]; then
echo "changes=true" >> $GITHUB_OUTPUT
else
echo "changes=false" >> $GITHUB_OUTPUT
fi
- name: Commit Quality Fixes
if: steps.check-quality-changes.outputs.changes == 'true'
run: |
git add packages/docs/
git commit -m "fix(docs): automated documentation quality improvements [skip ci]" || true
# ===========================================
# Create PR with all fixes
# ===========================================
create-pr:
needs: [check-links, check-quality]
if: always() && (needs.check-links.result == 'success' || needs.check-quality.result == 'success')
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- name: Checkout repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 0
filter: blob:none
token: ${{ secrets.GITHUB_TOKEN }}
- name: Check for any uncommitted changes
id: final-check
run: |
git fetch origin
if [ -n "$(git status --porcelain packages/docs/)" ]; then
echo "changes=true" >> $GITHUB_OUTPUT
else
echo "changes=false" >> $GITHUB_OUTPUT
fi
- name: Create Fix Branch and PR
if: steps.final-check.outputs.changes == 'true'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
BRANCH_NAME="fix/docs-ci-$(date +%Y%m%d-%H%M%S)"
git checkout -b "$BRANCH_NAME"
git add packages/docs/
git commit -m "fix(docs): automated documentation fixes" \
-m "- Fixed broken links" \
-m "- Fixed quality issues" || exit 0
git push origin "$BRANCH_NAME"
gh pr create \
--title "fix(docs): automated documentation fixes" \
--body "## Automated Documentation Fixes
This PR contains automated fixes for documentation issues.
### Changes Made:
- 🔗 Fixed broken links
- 📝 Fixed quality issues (double headers, missing frontmatter, etc.)
### Review Instructions:
Please verify all changes are correct before merging.
---
*Generated by docs-ci workflow*" \
--base main \
--head "$BRANCH_NAME" \
--label "documentation" \
--label "automated"
@@ -0,0 +1,81 @@
# Electrobun Submodule Guard
#
# Fails fast when the `upstreams/electrobun` submodule gitlink points at a commit
# that is NOT fetchable from its configured remote — the recurring "dangling
# pointer" that breaks `git submodule update` for every fresh clone and CI run.
#
# History of this bug (each a commit that pinned an unfetchable SHA):
# db785c6dc6 -> fixed by 2881d83894 -> regressed to 6bedce8261 (evil merge
# f5ae5dfed5) -> root cause finally fixed by d32749acdc, which repointed the
# submodule URL at the elizaOS/electrobun fork (where the real, ahead-of-public
# electrobun development actually lives) instead of upstream blackboardsh/electrobun.
#
# This guard keeps it from regressing: any PR/push that moves the gitlink or the
# URL must leave a pin that the remote can actually serve.
name: Electrobun Submodule Guard
on:
pull_request:
paths:
- "upstreams/electrobun"
- ".gitmodules"
push:
branches: [develop, main]
paths:
- "upstreams/electrobun"
- ".gitmodules"
# cancel superseded in-flight runs for the same PR to free hosted-runner
# capacity. only cancels pull_request runs; push runs on protected branches
# (main/develop) are never cancelled mid-flight.
# CI fan-out fix: the PR-scoped concurrency from #14069 fell back to
# github.run_id on push, so every develop push got a unique group and
# nothing ever superseded. A 37-merge wave queued thousands of push runs
# and starved the self-hosted fleet. Fall back to github.ref (all develop
# pushes share a group) and cancel superseded runs on push too. PR behavior
# is unchanged (pull_request.number still wins the key); merge_group and
# schedule events are not in the cancel set, so the merge queue and nightly
# runs always complete.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }}
permissions:
contents: read
jobs:
verify-electrobun-pin:
name: electrobun gitlink is fetchable
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 5
steps:
- name: Checkout (no submodules)
# actions/checkout@v4
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
with:
submodules: false
- name: Verify the electrobun pin is fetchable from its remote
run: |
set -euo pipefail
url=$(git config -f .gitmodules submodule.upstreams/electrobun.url || true)
sha=$(git ls-tree HEAD upstreams/electrobun | awk '{print $3}' || true)
echo "configured url: ${url:-<none>}"
echo "pinned commit: ${sha:-<none>}"
if [ -z "${url}" ] || [ -z "${sha}" ]; then
echo "::error::Could not read the electrobun submodule URL or pinned commit from .gitmodules / the tree."
exit 1
fi
# Mirror exactly what `git submodule update` does: fetch that exact commit.
# GitHub serves any commit reachable from a ref (allowReachableSHA1InWant);
# a dangling/unpushed commit fails here just as it would for a fresh clone.
tmp=$(mktemp -d)
git init -q "${tmp}"
if git -C "${tmp}" fetch -q --depth=1 "${url}" "${sha}"; then
echo "OK: ${sha} is fetchable from ${url}"
else
echo "::error::electrobun submodule pin ${sha} is NOT fetchable from ${url}."
echo "::error::A fresh 'git submodule update' will fail ('not our ref') — this is the recurring dangling-gitlink bug."
echo "::error::Fix: push that electrobun commit to the configured remote, or repoint the pin/URL to a commit the remote serves (e.g. the elizaOS/electrobun develop or main HEAD)."
exit 1
fi
+142
View File
@@ -0,0 +1,142 @@
name: ElizaOS Cuttlefish
# Brand-aware AOSP / Cuttlefish build pipeline.
#
# Defaults to the elizaOS brand (packages/scripts/distro-android/brand.eliza.json).
# Downstream brands (e.g. ElizaOS) supply their own brand config + vendor
# tree via `brand-config` and `vendor-source` inputs and call this workflow
# either via workflow_dispatch (manual run from this repo) or via
# workflow_call (cross-repo reusable workflow):
#
# # downstream/.github/workflows/my-brand-cuttlefish.yml
# jobs:
# build:
# uses: elizaOS/eliza/.github/workflows/elizaos-cuttlefish.yml@develop
# with:
# brand-config: packages/os/android/brand.mybrand.json
# vendor-source: packages/os/android/vendor/mybrand
# aosp-root: /aosp
# launch: true
on:
workflow_dispatch:
inputs:
brand-config:
description: "Path to brand config JSON (relative to repo root). Default: eliza."
required: false
type: string
default: "packages/scripts/distro-android/brand.eliza.json"
vendor-source:
description: "Override path to the vendor tree (relative to repo root). Defaults to brand.vendorDir."
required: false
type: string
default: ""
aosp-root:
description: "Existing AOSP checkout root on the Linux/KVM runner"
required: true
type: string
jobs:
description: "AOSP build parallelism"
required: false
default: "8"
type: string
launch:
description: "Launch Cuttlefish and run adb boot validation"
required: false
default: true
type: boolean
workflow_call:
inputs:
brand-config:
description: "Path to brand config JSON (relative to repo root)."
required: false
type: string
default: "packages/scripts/distro-android/brand.eliza.json"
vendor-source:
description: "Override path to the vendor tree."
required: false
type: string
default: ""
aosp-root:
description: "Existing AOSP checkout root."
required: true
type: string
jobs:
description: "AOSP build parallelism."
required: false
type: string
default: "8"
launch:
description: "Launch Cuttlefish and run adb boot validation."
required: false
type: boolean
default: true
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
build-and-validate:
name: Build and validate brand AOSP image
runs-on: [self-hosted, linux, x64, kvm]
timeout-minutes: 720
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: recursive
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: canary
- uses: actions/setup-java@ad2b38190b15e4d6bdf0c97fb4fca8412226d287
with:
distribution: temurin
java-version: "21"
# The brand validator shells out to `aapt` to inspect the staged APK,
# and the brand `build:android:system` runs Gradle which needs a full SDK.
- uses: android-actions/setup-android@40fd30fb8d7440372e1316f5d1809ec01dcd3699
with:
packages: "platform-tools build-tools;36.0.0 platforms;android-36"
- name: Install workspace dependencies
run: bun install --frozen-lockfile || bun install --no-frozen-lockfile
- name: Resolve brand
id: brand
run: |
BRAND_CONFIG="${{ inputs.brand-config }}"
if [ ! -f "$BRAND_CONFIG" ]; then
echo "::error::Brand config not found: $BRAND_CONFIG"
exit 1
fi
echo "config=$BRAND_CONFIG" >> "$GITHUB_OUTPUT"
BUILD_CMD=$(node -e "const c=JSON.parse(require('fs').readFileSync('$BRAND_CONFIG','utf8')); console.log((c.buildAndroidSystemCmd ?? ['bun','run','build:android:system']).join(' '))")
echo "build-cmd=$BUILD_CMD" >> "$GITHUB_OUTPUT"
DISTRO_NAME=$(node -p "JSON.parse(require('fs').readFileSync('$BRAND_CONFIG','utf8')).distroName")
echo "distro-name=$DISTRO_NAME" >> "$GITHUB_OUTPUT"
- name: Build privileged APK (${{ steps.brand.outputs.distro-name }})
run: ${{ steps.brand.outputs.build-cmd }}
- name: Static brand validation
run: |
node packages/scripts/distro-android/validate.mjs \
--brand-config "${{ steps.brand.outputs.config }}"
- name: Sync, build, and optionally boot
run: |
ARGS=(
--brand-config "${{ steps.brand.outputs.config }}"
--aosp-root "${{ inputs.aosp-root }}"
--jobs "${{ inputs.jobs }}"
)
if [ -n "${{ inputs.vendor-source }}" ]; then
ARGS+=(--source-vendor "${{ inputs.vendor-source }}")
fi
if [ "${{ inputs.launch }}" = "true" ]; then
ARGS+=(--launch --boot-validate)
fi
node packages/scripts/distro-android/build-aosp.mjs "${ARGS[@]}"
@@ -0,0 +1,290 @@
name: elizaOS Full OS Release
on:
release:
types: [created]
workflow_dispatch:
inputs:
publish:
description: Upload artifacts to release
type: boolean
default: false
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
# Compute the bare semver version string (no leading 'v') from the ref name.
# On a 'release' event github.ref_name is the tag (e.g. v2.0.1); we strip
# the prefix here so downstream jobs get a clean version like '2.0.1'.
prepare:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
outputs:
version: ${{ steps.parse.outputs.version }}
tag: ${{ steps.parse.outputs.tag }}
steps:
- name: Parse version from ref
id: parse
run: |
TAG="${{ github.ref_name }}"
VERSION="${TAG#v}"
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
echo "Resolved: tag=$TAG version=$VERSION"
trigger-linux-iso:
uses: ./.github/workflows/build-linux-iso.yml
with:
channel: stable
publish: ${{ github.event_name == 'release' || inputs.publish == true }}
secrets: inherit
trigger-debian-package:
uses: ./.github/workflows/build-debian-package.yml
secrets: inherit
trigger-vm-image:
uses: ./.github/workflows/build-vm-image.yml
with:
format: ova
publish: ${{ github.event_name == 'release' || inputs.publish == true }}
secrets: inherit
verify-artifacts:
name: Verify build artifacts exist and are non-empty
needs: [trigger-linux-iso, trigger-debian-package, trigger-vm-image]
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 10
steps:
- name: Download Linux ISO artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c
with:
pattern: elizaos-live-iso-*
path: ./_verify/iso
merge-multiple: true
- name: Download Debian package artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c
with:
name: elizaos-debian-package
path: ./_verify/deb
- name: Download VM image artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c
with:
name: elizaos-vm-image
path: ./_verify/vm
- name: Assert artifacts present and non-empty
run: |
set -euo pipefail
fail=0
assert_nonempty() {
local pattern="$1"
local label="$2"
local match
match=$(find "$pattern" -type f \( -name "$3" \) -size +0c 2>/dev/null | head -1)
if [ -z "$match" ]; then
echo "ERROR: missing or empty $label artifact (pattern $3 under $pattern)"
fail=1
else
echo "OK: $label -> $match ($(stat --format=%s "$match") bytes)"
fi
}
assert_nonempty ./_verify/iso "Linux ISO" "*.iso"
assert_nonempty ./_verify/deb "Debian .deb" "*.deb"
assert_nonempty ./_verify/vm "VM image" "*.ova"
exit "$fail"
populate-and-validate-manifest:
name: Populate manifest checksums and gate on publishable validation
needs: [verify-artifacts]
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 15
permissions:
# contents: write — required by softprops/action-gh-release to
# attach SHA256SUMS to the release asset list.
# id-token: write + attestations: write — required by
# actions/attest-build-provenance@v4 to mint a Sigstore-signed
# SLSA provenance attestation over SHA256SUMS via GitHub OIDC.
contents: write
id-token: write
attestations: write
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: "canary"
- name: Download all release artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c
with:
path: ./_artifacts
merge-multiple: true
- name: Check manifest file exists
id: manifest_check
# The manifest file is hand-maintained per beta cycle. If it is
# missing (or got renamed without this workflow being updated)
# every downstream step would fail with ENOENT inside a Node
# readFileSync — a confusing 1-2 min failure with no clear call
# to action. Surface it explicitly so the maintainer either adds
# the manifest, updates this hardcoded path, or accepts the skip.
env:
MANIFEST_PATH: packages/os/release/beta-2026-05-16/manifest.json
run: |
if [ -f "$MANIFEST_PATH" ]; then
echo "can_validate_manifest=true" >> "$GITHUB_OUTPUT"
else
echo "::warning::Release manifest $MANIFEST_PATH not found. Manifest population + canonical SHA256SUMS + SLSA-over-SHA256SUMS attestation skipped. Per-artifact SLSA attestations from build-linux-iso.yml / build-debian-package.yml / build-vm-image.yml still mint correctly."
echo "can_validate_manifest=false" >> "$GITHUB_OUTPUT"
fi
- name: Populate manifest with real sha256 + sizes
if: steps.manifest_check.outputs.can_validate_manifest == 'true'
# Discovery is driven by the *manifest filename* — each entry's
# `filename` is searched verbatim under ./_artifacts. This keeps
# the workflow and manifest in lock-step: rename in the manifest,
# rename in the build, no workflow edits required.
#
# Build workflows that currently feed this step:
# - build-linux-iso.yml → *.iso (raw-linux-* entry)
# - build-vm-image.yml → *.qcow2, *.ova (vm-linux-* entry)
# Android (cf_x86_64, pixel-arm64) and macOS UTM bundles are NOT
# produced by any CI workflow today. They must be attached to the
# _artifacts staging dir out-of-band (release engineer uploads the
# built bundle as a workflow artifact via a prior manual job, or
# via release asset download) before this job runs. The strict
# gate at the end will fail loudly if any artifact is missing.
run: |
set -euo pipefail
MANIFEST=packages/os/release/beta-2026-05-16/manifest.json
ids=$(node -e '
const m = JSON.parse(require("node:fs").readFileSync(process.argv[1], "utf8"));
for (const a of m.artifacts) console.log(`${a.id}\t${a.filename}`);
' "$MANIFEST")
while IFS=$'\t' read -r id filename; do
path=$(find ./_artifacts -type f -name "$filename" | head -1 || true)
if [ -z "$path" ] || [ ! -f "$path" ]; then
echo "::warning::no artifact named $filename found for $id — strict gate will fail"
continue
fi
sha=$(sha256sum "$path" | cut -d' ' -f1)
size=$(stat --format=%s "$path")
node packages/os/scripts/update-release-manifest.mjs \
--manifest "$MANIFEST" \
--artifact "$id" \
--sha256 "$sha" \
--size "$size"
done <<< "$ids"
- name: Gate on publishable checksums (before status promotion)
if: steps.manifest_check.outputs.can_validate_manifest == 'true'
# Status promotion is gated on strict validation. If any artifact
# is missing a real checksum, the release stays "candidate" and
# this job fails — no half-published manifest.
run: |
node packages/os/scripts/validate-release-manifest.mjs \
--manifest packages/os/release/beta-2026-05-16/manifest.json \
--require-publishable-checksums
- name: Generate canonical SHA256SUMS via the in-repo tool
if: steps.manifest_check.outputs.can_validate_manifest == 'true'
# Runs BEFORE manifest promotion so that any failure here prevents
# the "available" status from being set and uploaded. SHA256SUMS
# generation is a hard prerequisite for promotion — closing the
# window where a promoted manifest could exist without a
# corresponding SHA256SUMS file or SLSA attestation.
#
# Calls packages/os/scripts/generate-release-checksums.mjs (which
# already exists, has unit tests under __tests__, and is the
# repo's canonical checksum tool) against the populated manifest
# + downloaded artifacts. Produces sha256sum -c compatible output
# mirroring the Debian / Fedora / Tails / Arch convention.
run: |
set -euo pipefail
node packages/os/scripts/generate-release-checksums.mjs \
--manifest packages/os/release/beta-2026-05-16/manifest.json \
--artifact-root ./_artifacts \
--output ./_artifacts/SHA256SUMS
echo "Generated SHA256SUMS:"
cat ./_artifacts/SHA256SUMS
- name: Attest SLSA build provenance over SHA256SUMS
if: steps.manifest_check.outputs.can_validate_manifest == 'true'
# Mints a single Sigstore-signed in-toto/SLSA provenance
# attestation that covers every release artifact via its hash,
# by attesting the canonical SHA256SUMS file generated above.
# Requires id-token: write + attestations: write (set in this
# job's permissions block). Users verify with:
#
# gh attestation verify SHA256SUMS --owner elizaOS
# sha256sum -c SHA256SUMS --ignore-missing
uses: actions/attest-build-provenance@0f67c3f4856b2e3261c31976d6725780e5e4c373
with:
subject-path: ./_artifacts/SHA256SUMS
- name: Upload SHA256SUMS artifact
if: steps.manifest_check.outputs.can_validate_manifest == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: elizaos-release-checksums
path: ./_artifacts/SHA256SUMS
retention-days: 90
- name: Promote release status to available
if: steps.manifest_check.outputs.can_validate_manifest == 'true'
run: |
set -euo pipefail
node -e '
const fs = require("node:fs");
const p = "packages/os/release/beta-2026-05-16/manifest.json";
const m = JSON.parse(fs.readFileSync(p, "utf8"));
m.release.status = "available";
fs.writeFileSync(p, JSON.stringify(m, null, 2) + "\n");
'
- name: Upload populated manifest
if: steps.manifest_check.outputs.can_validate_manifest == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: elizaos-os-release-manifest-published
path: packages/os/release/beta-2026-05-16/manifest.json
retention-days: 90
- name: Attach SHA256SUMS to release
if: github.event_name == 'release' && steps.manifest_check.outputs.can_validate_manifest == 'true'
uses: softprops/action-gh-release@718ea10b132b3b2eba29c1007bb80653f286566b
with:
files: ./_artifacts/SHA256SUMS
tag_name: ${{ github.event.release.tag_name || github.ref_name }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
publish-apt:
needs: [prepare, trigger-debian-package, verify-artifacts]
if: needs.trigger-debian-package.result == 'success' && needs.verify-artifacts.result == 'success'
uses: ./.github/workflows/publish-apt-repo.yml
with:
version: ${{ needs.prepare.outputs.version }}
tag: ${{ needs.prepare.outputs.tag }}
channel: stable
secrets: inherit
summary:
needs: [trigger-linux-iso, trigger-debian-package, trigger-vm-image, verify-artifacts, populate-and-validate-manifest, publish-apt]
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
if: always()
timeout-minutes: 5
steps:
- name: Release summary
run: |
{
echo "## elizaOS OS Release Summary"
echo "| Artifact | Status |"
echo "| -------- | ------ |"
echo "| Linux ISO | ${{ needs.trigger-linux-iso.result }} |"
echo "| Debian .deb | ${{ needs.trigger-debian-package.result }} |"
echo "| VM Image | ${{ needs.trigger-vm-image.result }} |"
echo "| Verify artifacts | ${{ needs.verify-artifacts.result }} |"
echo "| Populate + gate manifest | ${{ needs.populate-and-validate-manifest.result }} |"
echo "| apt Repo | ${{ needs.publish-apt.result }} |"
} >> "$GITHUB_STEP_SUMMARY"
+131
View File
@@ -0,0 +1,131 @@
name: ElizaOS OS release validation
on:
workflow_dispatch:
pull_request:
branches: [main]
paths:
- ".github/workflows/elizaos-os-release.yml"
- "packages/os/**"
- "packages/os/homepage/**"
- "packages/os/usb-installer/**"
- "packages/os/setup/**"
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
validate-os-release:
name: Validate OS release surfaces
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Set up Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: "canary"
- name: Install dependencies
run: bun install
- name: Validate OS release manifest
run: |
manifest=packages/os/release/beta-2026-05-16/manifest.json
evidence=packages/os/release/beta-2026-05-16/evidence/ci-evidence.json
node --check packages/os/scripts/*.mjs
node --test packages/os/scripts/__tests__/os-release-scripts.test.mjs
if [ -f "$manifest" ]; then
node packages/os/scripts/validate-release-manifest.mjs \
--manifest "$manifest"
node packages/os/scripts/collect-release-evidence.mjs \
--manifest "$manifest" \
--output "$evidence"
else
echo "::notice::${manifest} is not present; skipping beta release manifest validation."
fi
- name: Validate USB installer
run: |
bun run --cwd packages/os/usb-installer lint
bun run --cwd packages/os/usb-installer typecheck
bun run --cwd packages/os/usb-installer test
bun run --cwd packages/os/usb-installer build
bunx playwright install --with-deps chromium
bun run --cwd packages/os/usb-installer test:e2e
if modinfo scsi_debug >/dev/null 2>&1; then
bun run --cwd packages/os/usb-installer test:linux-virtual-usb
else
echo "::notice::scsi_debug kernel module is unavailable on this runner; skipping virtual block-device USB proof."
fi
- name: Validate Android installer
run: |
android_manifest=packages/os/release/beta-2026-05-16/android-release-manifest.json
packages/os/android/installer/tests/run-tests.sh
node packages/os/android/installer/scripts/validate-release-manifest.mjs \
packages/os/android/installer/manifests/android-release-manifest.example.json
if [ -f "$android_manifest" ]; then
node packages/os/android/installer/scripts/validate-release-manifest.mjs \
"$android_manifest" \
--allow-placeholders
else
echo "::notice::${android_manifest} is not present; skipping beta Android release manifest validation."
fi
- name: Validate elizaos-setup (formerly aosp-flasher)
run: |
bun run --cwd packages/os/setup typecheck
bun run --cwd packages/os/setup build
- name: Validate Linux live USB metadata
run: |
cd packages/os/linux
ELIZAOS_STATIC_SOURCE_ONLY=1 ./scripts/static-smoke.sh
- name: Validate OS homepage
run: |
bun run --cwd packages/os/homepage lint:check
bun run --cwd packages/os/homepage typecheck
bun run --cwd packages/os/homepage build
(cd packages/os/homepage && ../../node_modules/.bin/playwright install --with-deps chromium)
bun run --cwd packages/os/homepage test:e2e
- name: Capture homepage contact sheet
run: |
set -euo pipefail
bun --bun --cwd packages/os/homepage vite preview --host 127.0.0.1 --port 4455 &
echo $! > /tmp/os-homepage-preview.pid
ready=0
for attempt in {1..60}; do
if curl -fsS http://127.0.0.1:4455 >/dev/null; then
ready=1
break
fi
echo "Waiting for os-homepage preview (${attempt}/60)..."
sleep 1
done
if [ "$ready" != "1" ]; then
echo "ERROR: os-homepage preview never came up on 127.0.0.1:4455"
kill "$(cat /tmp/os-homepage-preview.pid)" 2>/dev/null || true
exit 1
fi
bun run --cwd packages/os/homepage screenshots
kill "$(cat /tmp/os-homepage-preview.pid)" 2>/dev/null || true
- name: Upload release evidence
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
if: always()
with:
name: elizaos-os-release-validation
path: |
packages/os/release/beta-2026-05-16/evidence/**
packages/os/linux/vm/output/ci-bundle-metadata/**
packages/os/homepage/artifacts/screenshots/**
packages/os/homepage/test-results/**
if-no-files-found: warn
retention-days: 14
@@ -0,0 +1,137 @@
name: External API Live Drift
# Runs the external-API LIVE-DRIFT tests against the real provider APIs so the
# keyless mocks/parsers are continuously verified to still match reality. These
# *.real.test.ts files self-skip in the PR lane (their LIVE gate is off) and ran
# in NO workflow until now — they were dark. This lane turns the LIVE gate on and
# injects the optional provider secrets.
#
# Provider coverage vs provider-coverage.ts (#8801):
# PUBLIC (no secret, always run): Polymarket, Hyperliquid, CoinGecko.
# KEY/TOKEN-GATED (self-skip via describe.skipIf unless their
# <PROVIDER>_API_KEY / _ACCESS_TOKEN secret is present): Calendly, Tavily,
# OpenAI, Anthropic, Strava, Oura, Withings, Fitbit.
# CONFIG-BLOCKED (test exists + secret wired, but the plugin's vitest config
# excludes *.real.test.ts so it can't run from here yet): Google Calendar —
# see the structural-lane marker at the bottom of the steps.
# A missing secret is a clean no-op (describe.skipIf), not a failure — so the
# credential-blocked lanes only light up once their token secret is added to the
# repo. continue-on-error keeps a provider outage / drift from red-X'ing the
# branch — a failure here is a real signal that a mock has drifted from the live
# API. The MTProto/native surfaces (telegram-mtproto, signal,
# imessage-bluebubbles) have no HTTP drift surface and stay hand-curated
# mock-only with that caveat documented in
# packages/test/mocks/helpers/provider-coverage.ts.
#
# Nightly + on-demand; never on pull_request.
on:
schedule:
# 09:30 UTC daily — after the other nightly live lanes settle.
- cron: "30 9 * * *"
workflow_dispatch: {}
concurrency:
group: external-api-live-drift-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
CI: "true"
NODE_VERSION: "24"
BUN_VERSION: "canary"
NODE_OPTIONS: "--experimental-sqlite"
# Turn the LIVE gate on for every external-API drift test.
POLYMARKET_LIVE_TEST: "1"
HYPERLIQUID_LIVE_TEST: "1"
COINGECKO_LIVE_TEST: "1"
CALENDLY_LIVE_TEST: "1"
TAVILY_LIVE_TEST: "1"
OPENAI_LIVE_TEST: "1"
ANTHROPIC_LIVE_TEST: "1"
STRAVA_LIVE_TEST: "1"
OURA_LIVE_TEST: "1"
WITHINGS_LIVE_TEST: "1"
FITBIT_LIVE_TEST: "1"
GOOGLE_CALENDAR_LIVE_TEST: "1"
# Key/token-gated providers (tests self-skip when their secret is absent).
CALENDLY_ACCESS_TOKEN: ${{ secrets.CALENDLY_ACCESS_TOKEN }}
TAVILY_API_KEY: ${{ secrets.TAVILY_API_KEY }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
STRAVA_ACCESS_TOKEN: ${{ secrets.STRAVA_ACCESS_TOKEN }}
OURA_ACCESS_TOKEN: ${{ secrets.OURA_ACCESS_TOKEN }}
WITHINGS_ACCESS_TOKEN: ${{ secrets.WITHINGS_ACCESS_TOKEN }}
FITBIT_ACCESS_TOKEN: ${{ secrets.FITBIT_ACCESS_TOKEN }}
GOOGLE_CALENDAR_ACCESS_TOKEN: ${{ secrets.GOOGLE_CALENDAR_ACCESS_TOKEN }}
jobs:
live-drift:
name: External-API mocks vs live APIs (nightly)
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 30
continue-on-error: true
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
submodules: false
show-progress: false
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e
with:
node-version: ${{ env.NODE_VERSION }}
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: ${{ env.BUN_VERSION }}
- name: Install dependencies
run: bun install --frozen-lockfile
- name: Polymarket live drift (public)
run: bunx vitest run --root plugins/plugin-polymarket src/routes.real.test.ts
- name: Hyperliquid live drift (public)
run: bunx vitest run --root plugins/plugin-hyperliquid src/routes.real.test.ts
- name: CoinGecko live drift (public)
run: bunx vitest run --root plugins/plugin-wallet src/routes/wallet-market-overview.real.test.ts
- name: Calendly live drift (token-gated, self-skips)
run: bunx vitest run --root plugins/plugin-calendly src/calendly-client.real.test.ts
- name: Tavily web-search live drift (key-gated, self-skips)
run: bunx vitest run --root plugins/plugin-web-search src/services/webSearchService.real.test.ts
- name: OpenAI live drift (key-gated, self-skips)
run: bunx vitest run --root plugins/plugin-openai __tests__/openai-drift.real.test.ts
- name: Anthropic live drift (key-gated, self-skips)
run: bunx vitest run --root plugins/plugin-anthropic __tests__/anthropic-drift.real.test.ts
- name: Strava live drift (token-gated, self-skips)
run: bunx vitest run --root plugins/plugin-health test/strava-connector.real.test.ts
- name: Oura live drift (token-gated, self-skips)
run: bunx vitest run --root plugins/plugin-health test/oura-connector.real.test.ts
- name: Withings live drift (token-gated, self-skips)
run: bunx vitest run --root plugins/plugin-health test/withings-connector.real.test.ts
- name: Fitbit live drift (token-gated, self-skips)
run: bunx vitest run --root plugins/plugin-health test/fitbit-connector.real.test.ts
# Structural-lane marker — NOT YET RUNNABLE here.
# plugins/plugin-calendar/test/google-calendar-connector.real.test.ts
# exists and self-gates on GOOGLE_CALENDAR_LIVE_TEST + GOOGLE_CALENDAR_ACCESS_TOKEN
# (secret wired in the env block above), but plugin-calendar's vitest.config.ts
# excludes `**/*.real.test.{ts,tsx}` with passWithNoTests:true, so a vitest
# invocation here finds zero tests and would silently "pass" with no drift
# coverage. That config exclude can only be lifted in the plugin's own config.
# Wire the step below once plugin-calendar's config stops excluding real tests:
# run: bunx vitest run --root plugins/plugin-calendar test/google-calendar-connector.real.test.ts
@@ -0,0 +1,57 @@
name: Apply database migrations
on:
push:
branches:
- staging
- production
concurrency:
group: apply-migrations-${{ github.ref }}
cancel-in-progress: false
permissions:
contents: read
jobs:
migrate:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
env:
STAGING_DATABASE_URL: ${{ secrets.STAGING_DATABASE_URL }}
STAGING_DIRECT_DATABASE_URL: ${{ secrets.STAGING_DIRECT_DATABASE_URL }}
PROD_DATABASE_URL: ${{ secrets.PROD_DATABASE_URL }}
PROD_DIRECT_DATABASE_URL: ${{ secrets.PROD_DIRECT_DATABASE_URL }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: canary
- name: Install dependencies
run: bun install --frozen-lockfile
- name: Apply migrations (staging)
if: github.ref == 'refs/heads/staging' && env.STAGING_DATABASE_URL != '' && env.STAGING_DIRECT_DATABASE_URL != ''
env:
DATABASE_URL: ${{ env.STAGING_DATABASE_URL }}
DIRECT_DATABASE_URL: ${{ env.STAGING_DIRECT_DATABASE_URL }}
working-directory: packages/feed/packages/db
run: bunx --bun drizzle-kit push --force --config=drizzle.config.ts
- name: Apply migrations (production)
if: github.ref == 'refs/heads/production' && env.PROD_DATABASE_URL != '' && env.PROD_DIRECT_DATABASE_URL != ''
env:
DATABASE_URL: ${{ env.PROD_DATABASE_URL }}
DIRECT_DATABASE_URL: ${{ env.PROD_DIRECT_DATABASE_URL }}
working-directory: packages/feed/packages/db
run: bunx --bun drizzle-kit push --force --config=drizzle.config.ts
- name: Skip notice (staging)
if: github.ref == 'refs/heads/staging' && (env.STAGING_DATABASE_URL == '' || env.STAGING_DIRECT_DATABASE_URL == '')
run: echo "Skipping staging migrations because STAGING_DATABASE_URL or STAGING_DIRECT_DATABASE_URL is not set."
- name: Skip notice (production)
if: github.ref == 'refs/heads/production' && (env.PROD_DATABASE_URL == '' || env.PROD_DIRECT_DATABASE_URL == '')
run: echo "Skipping production migrations because PROD_DATABASE_URL or PROD_DIRECT_DATABASE_URL is not set."
+46
View File
@@ -0,0 +1,46 @@
name: Env Audit
on:
pull_request:
push:
branches: [develop]
# CI fan-out fix: the PR-scoped concurrency from #14069 fell back to
# github.run_id on push, so every develop push got a unique group and
# nothing ever superseded. A 37-merge wave queued thousands of push runs
# and starved the self-hosted fleet. Fall back to github.ref (all develop
# pushes share a group) and cancel superseded runs on push too. PR behavior
# is unchanged (pull_request.number still wins the key); merge_group and
# schedule events are not in the cancel set, so the merge queue and nightly
# runs always complete.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }}
env:
CI: true
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
env-audit:
name: env:audit:check
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 2
- name: Setup Bun workspace
uses: ./.github/actions/setup-bun-workspace
with:
bun-version: "1.3.14" # pinned: floating canary writes lockfileVersion 2 and breaks --frozen-lockfile (#11184/#9454)
install-command: bun install
- name: Env audit check
run: bun run --cwd packages/feed env:audit:check
continue-on-error: true
+423
View File
@@ -0,0 +1,423 @@
name: Feed RL Training
# Trigger options:
# 1. On schedule (daily training at 2 AM UTC)
# 2. Via repository_dispatch (from debug endpoint)
# 3. Manual workflow_dispatch (testing from GitHub UI)
on:
schedule:
# Daily at 2 AM UTC
- cron: '0 2 * * *'
repository_dispatch:
types: [trigger-training]
workflow_dispatch:
inputs:
batch_id:
description: 'Training batch ID (optional, auto-generated if not provided)'
required: false
type: string
window_id:
description: 'Window ID to train (optional, auto-detected if not provided)'
required: false
type: string
force:
description: 'Force training even if not ready (for testing)'
required: false
type: boolean
default: false
base_model:
description: 'Base model to use (default: google/gemma-4-E4B-it)'
required: false
type: string
env:
PYTHON_VERSION: '3.11'
FEED_DEFAULT_BASE_MODEL: 'google/gemma-4-E4B-it'
# Prevent concurrent training runs
concurrency:
group: training-pipeline
cancel-in-progress: false # Wait for current run to finish
permissions:
contents: read
jobs:
train:
if: ${{ vars.ENABLE_RL_TRAINING == 'true' }}
name: Train RL Model
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 360 # 6 hours max
steps:
- name: Checkout code
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1
with:
python-version: ${{ env.PYTHON_VERSION }}
cache: 'pip'
cache-dependency-path: 'packages/training/scripts/rl/requirements.txt'
- name: Install dependencies
working-directory: packages/training/scripts/rl
run: |
pip install --upgrade pip
pip install -r requirements.txt
pip install -e .
- name: Verify installation
run: |
python --version
pip list | grep -E "(art|asyncpg)"
- name: Check training readiness
id: readiness
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
run: |
python -c "
import asyncio
import asyncpg
import os
import sys
async def check():
try:
pool = await asyncpg.create_pool(
os.getenv('DATABASE_URL'),
min_size=1,
max_size=2,
timeout=30
)
# Count scored trajectories ready for training
count = await pool.fetchval('''
SELECT COUNT(*) FROM trajectories
WHERE \"isTrainingData\" = true
AND \"usedInTraining\" = false
AND \"aiJudgeReward\" IS NOT NULL
AND \"stepsJson\" IS NOT NULL
AND \"stepsJson\"::text != 'null'
AND \"stepsJson\"::text != '[]'
''')
print(f'✅ Database connected')
print(f'📊 Trajectories ready for training: {count}')
# Need 100 bundles minimum
min_required = 100
ready = count >= min_required
if ready:
print(f'✅ READY: {count} >= {min_required}')
else:
print(f'⏳ NOT READY: {count} < {min_required} (need {min_required - count} more)')
with open(os.getenv('GITHUB_OUTPUT'), 'a') as f:
f.write(f'ready={str(ready).lower()}\\n')
f.write(f'count={count}\\n')
await pool.close()
except Exception as e:
print(f'❌ Error checking readiness: {e}', file=sys.stderr)
with open(os.getenv('GITHUB_OUTPUT'), 'a') as f:
f.write('ready=false\\n')
f.write('count=0\\n')
sys.exit(1)
asyncio.run(check())
"
- name: Get batch info
id: batch
env:
# From workflow_dispatch inputs:
BATCH_ID_INPUT: ${{ inputs.batch_id || '' }}
WINDOW_ID_INPUT: ${{ inputs.window_id || '' }}
FORCE_INPUT: ${{ inputs.force || 'false' }}
BASE_MODEL_INPUT: ${{ inputs.base_model || '' }}
# From repository_dispatch payload:
BATCH_ID_PAYLOAD: ${{ github.event.client_payload.batch_id || '' }}
WINDOW_ID_PAYLOAD: ${{ github.event.client_payload.window_id || '' }}
FORCE_PAYLOAD: ${{ github.event.client_payload.force || 'false' }}
DATABASE_URL: ${{ secrets.DATABASE_URL }}
run: |
# Determine batch_id
if [ -n "$BATCH_ID_PAYLOAD" ]; then
echo "batch_id=$BATCH_ID_PAYLOAD" >> $GITHUB_OUTPUT
elif [ -n "$BATCH_ID_INPUT" ]; then
echo "batch_id=$BATCH_ID_INPUT" >> $GITHUB_OUTPUT
else
echo "batch_id=batch-$(date +%s)" >> $GITHUB_OUTPUT
fi
# Determine window_id
if [ -n "$WINDOW_ID_PAYLOAD" ]; then
echo "window_id=$WINDOW_ID_PAYLOAD" >> $GITHUB_OUTPUT
elif [ -n "$WINDOW_ID_INPUT" ]; then
echo "window_id=$WINDOW_ID_INPUT" >> $GITHUB_OUTPUT
else
WINDOW=$(date -u +"%Y-%m-%dT%H:00")
echo "window_id=$WINDOW" >> $GITHUB_OUTPUT
fi
# Determine force flag
if [ "$FORCE_PAYLOAD" = "true" ] || [ "$FORCE_INPUT" = "true" ]; then
echo "force=true" >> $GITHUB_OUTPUT
else
echo "force=false" >> $GITHUB_OUTPUT
fi
# Determine base model
if [ -n "$BASE_MODEL_INPUT" ]; then
echo "base_model=$BASE_MODEL_INPUT" >> $GITHUB_OUTPUT
else
echo "base_model=$FEED_DEFAULT_BASE_MODEL" >> $GITHUB_OUTPUT
fi
# Generate model version
MODEL_VERSION=$(python -c "
import asyncio
import asyncpg
import os
import sys
async def get_version():
try:
pool = await asyncpg.create_pool(os.getenv('DATABASE_URL'), timeout=10)
latest = await pool.fetchval('''
SELECT version FROM trained_models
WHERE status IN ('ready', 'deployed')
ORDER BY \"createdAt\" DESC
LIMIT 1
''')
if latest:
parts = latest.strip('v').split('.')
patch = int(parts[2]) + 1
version = f'v{parts[0]}.{parts[1]}.{patch}'
else:
version = 'v1.0.0'
print(version)
await pool.close()
except Exception as e:
import time
version = f'v1.0.{int(time.time()) % 10000}'
print(version)
asyncio.run(get_version())
" 2>/dev/null)
echo "model_version=$MODEL_VERSION" >> $GITHUB_OUTPUT
echo "source=github_cron" >> $GITHUB_OUTPUT
- name: Skip if not ready (unless forced)
if: steps.readiness.outputs.ready != 'true' && steps.batch.outputs.force != 'true'
run: |
echo "⏭️ Not ready for training and force=false"
echo "Trajectories: ${{ steps.readiness.outputs.count }}"
echo "Required: 100 (minimum bundles)"
exit 0
- name: Update batch status to training
if: steps.readiness.outputs.ready == 'true' || steps.batch.outputs.force == 'true'
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
BATCH_ID: ${{ steps.batch.outputs.batch_id }}
run: |
python -c "
import asyncio
import asyncpg
import os
async def update():
pool = await asyncpg.create_pool(os.getenv('DATABASE_URL'))
batch_id = os.getenv('BATCH_ID')
async with pool.acquire() as conn:
async with conn.transaction():
await conn.execute('''
INSERT INTO training_batches (
\"batchId\", id, status, \"startedAt\", \"createdAt\"
) VALUES (
\$1, \$1, 'training', NOW(), NOW()
)
ON CONFLICT (\"batchId\")
DO UPDATE SET status = 'training', \"startedAt\" = NOW()
''', batch_id)
print(f'✅ Batch {batch_id} status: training')
await pool.close()
asyncio.run(update())
"
- name: Select base model
id: model_selection
if: steps.readiness.outputs.ready == 'true' || steps.batch.outputs.force == 'true'
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
BASE_MODEL_OVERRIDE: ${{ steps.batch.outputs.base_model }}
run: |
python -c "
import asyncio
import asyncpg
import os
import sys
async def select_model():
try:
pool = await asyncpg.create_pool(os.getenv('DATABASE_URL'))
base_model_override = os.getenv('BASE_MODEL_OVERRIDE', '')
default_base_model = os.getenv('FEED_DEFAULT_BASE_MODEL', 'google/gemma-4-E4B-it')
# Count training bundles
bundle_count = await pool.fetchval('''
SELECT COUNT(*) FROM trajectories
WHERE \"isTrainingData\" = true
AND \"usedInTraining\" = false
AND \"aiJudgeReward\" IS NOT NULL
AND \"stepsJson\" IS NOT NULL
AND \"stepsJson\"::text != 'null'
AND \"stepsJson\"::text != '[]'
''')
print(f'📊 Bundle count: {bundle_count}')
# Use override if provided, otherwise use default
if base_model_override:
base_model = base_model_override
strategy = 'override'
else:
base_model = default_base_model
strategy = 'default'
print(f'📦 Selected model: {base_model}')
print(f'📋 Strategy: {strategy}')
with open(os.getenv('GITHUB_OUTPUT'), 'a') as f:
f.write(f'base_model={base_model}\\n')
f.write(f'strategy={strategy}\\n')
f.write(f'bundle_count={bundle_count}\\n')
await pool.close()
except Exception as e:
print(f'❌ Model selection failed: {e}', file=sys.stderr)
default_base_model = os.getenv('FEED_DEFAULT_BASE_MODEL', 'google/gemma-4-E4B-it')
with open(os.getenv('GITHUB_OUTPUT'), 'a') as f:
f.write(f'base_model={default_base_model}\\n')
f.write('strategy=fallback\\n')
sys.exit(1)
asyncio.run(select_model())
"
- name: Run RL Training
id: training
if: steps.readiness.outputs.ready == 'true' || steps.batch.outputs.force == 'true'
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
BATCH_ID: ${{ steps.batch.outputs.batch_id }}
WINDOW_ID: ${{ steps.batch.outputs.window_id }}
MODEL_VERSION: ${{ steps.batch.outputs.model_version }}
BASE_MODEL: ${{ steps.model_selection.outputs.base_model }}
MODE: single
MAX_EXAMPLES: "2000"
MAX_STEPS_PER_TRAJECTORY: "20"
MAX_SEQ_LENGTH: "8192"
working-directory: packages/training/scripts/rl
run: |
echo "🚀 Starting training"
echo "Batch ID: $BATCH_ID"
echo "Window ID: $WINDOW_ID"
echo "Model Version: $MODEL_VERSION"
echo "Strategy: ${{ steps.model_selection.outputs.strategy }}"
echo "Base Model: $BASE_MODEL"
echo "Trajectories available: ${{ steps.readiness.outputs.count }}"
echo "Bundle count: ${{ steps.model_selection.outputs.bundle_count }}"
# Run trainer
python src/training/feed_trainer.py
- name: Update batch status to completed
if: success() && (steps.readiness.outputs.ready == 'true' || steps.batch.outputs.force == 'true')
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
BATCH_ID: ${{ steps.batch.outputs.batch_id }}
run: |
python -c "
import asyncio
import asyncpg
import os
async def update():
pool = await asyncpg.create_pool(os.getenv('DATABASE_URL'))
batch_id = os.getenv('BATCH_ID')
await pool.execute('''
UPDATE training_batches
SET status = 'completed', \"completedAt\" = NOW()
WHERE \"batchId\" = \$1
''', batch_id)
print(f'✅ Batch {batch_id} completed')
await pool.close()
asyncio.run(update())
"
- name: Update batch status to failed
if: failure() && (steps.readiness.outputs.ready == 'true' || steps.batch.outputs.force == 'true')
env:
DATABASE_URL: ${{ secrets.DATABASE_URL }}
BATCH_ID: ${{ steps.batch.outputs.batch_id }}
run: |
python -c "
import asyncio
import asyncpg
import os
async def update():
pool = await asyncpg.create_pool(os.getenv('DATABASE_URL'))
batch_id = os.getenv('BATCH_ID')
await pool.execute('''
UPDATE training_batches
SET status = 'failed', error = 'GitHub Actions workflow failed'
WHERE \"batchId\" = \$1
''', batch_id)
print(f'❌ Batch {batch_id} failed')
await pool.close()
asyncio.run(update())
"
- name: Upload training logs
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: training-logs-${{ steps.batch.outputs.batch_id }}
path: |
packages/training/scripts/rl/logs/
packages/training/scripts/rl/*.log
retention-days: 7
- name: Report status
if: always()
run: |
echo "Training Status: ${{ job.status }}"
echo "Batch ID: ${{ steps.batch.outputs.batch_id }}"
echo "Window ID: ${{ steps.batch.outputs.window_id }}"
echo "Ready: ${{ steps.readiness.outputs.ready }}"
echo "Trajectories: ${{ steps.readiness.outputs.count }}"
echo "Model: ${{ steps.model_selection.outputs.base_model }}"
+99
View File
@@ -0,0 +1,99 @@
name: feed-test
# Path-gated CI lane for the nested packages/feed monorepo (issue #9943).
#
# Why this exists: packages/feed has ~525 test files but ran in ZERO CI lanes.
# The root workspace excludes it (`!packages/feed`) and the root test runner
# honors that negation, while feed's own `test:ci` script was invoked by nothing.
# This lane runs feed's unit suite on every PR that touches packages/feed, so
# feed regressions are caught instead of silently shipping behind green CI.
#
# Scope decision (deliberate, documented): this runs `test:unit` — the isolated
# unit runner (scripts/test-unit-isolated.ts), which executes the bulk of feed's
# test files and explicitly excludes the suites that need infrastructure. It does
# NOT run the full `test:ci` script (scripts/ci/run-all-tests.sh): that script is
# a local-dev kitchen sink that also runs typecheck + lint + a production build,
# boots Anvil (foundry) + a server on :3000, and drives Playwright/Chroma E2E
# against a pre-seeded `.env.test`. Standing all of that up in a stock runner is
# fragile and would red-light every feed PR, so the heavier integration/E2E lane
# is a follow-up (provision postgres + foundry + the server + a CI `.env`). See
# the TODO below.
#
# TODO(#9943 follow-up): feed currently reports ~227 skipped tests. They need
# triage — some are env-gated (RUN_OPTIONAL_INTEGRATION_TESTS / RUN_LIVE_LLM_TESTS),
# some are stale. Audit and either wire the env or delete the skips, then promote
# this lane to also run `test:integration` (with a postgres service) and the E2E
# slice. Do NOT mass-unskip blindly.
on:
pull_request:
branches: [main]
paths:
- "packages/feed/**"
- ".github/workflows/feed-test.yml"
workflow_dispatch:
concurrency:
group: feed-test-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
env:
CI: "true"
jobs:
feed-unit:
name: feed unit tests
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
timeout-minutes: 25
# Required env for feed modules that validate configuration at import time.
# CI-only placeholders — no secrets. DATABASE_URL points at the postgres
# service below so any lazily-connecting module resolves a real DSN.
env:
DATABASE_URL: "postgres://postgres:postgres@localhost:5432/feed_test"
DIRECT_DATABASE_URL: "postgres://postgres:postgres@localhost:5432/feed_test"
NEXT_PUBLIC_STEWARD_API_URL: "http://localhost:31337"
STEWARD_JWT_SECRET: "ci-feed-unit-placeholder-jwt-secret"
CRON_SECRET: "ci-feed-unit-placeholder-cron-secret"
NODE_ENV: "test"
services:
postgres:
image: postgres:16
env:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: feed_test
ports:
- 5432:5432
options: >-
--health-cmd "pg_isready -U postgres"
--health-interval 10s
--health-timeout 5s
--health-retries 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: canary
# Install the repo root first so Bun links the root-owned elizaOS packages
# and the feed sub-workspaces that the root package.json globs. Then install
# the excluded feed root so its own app-level dependencies (drizzle-orm,
# Next, etc.) are available to `packages/feed/test:unit`. Feed postinstall
# bootstraps dev-only external agent frameworks, so skip scripts here.
- name: Install root workspace dependencies
run: bun install --frozen-lockfile
- name: Install feed root dependencies
working-directory: packages/feed
run: bun install --frozen-lockfile --ignore-scripts
- name: Build @elizaos/core dist
run: bun run --cwd packages/core build
- name: Run feed unit tests
working-directory: packages/feed
run: bun run test:unit
+104
View File
@@ -0,0 +1,104 @@
name: Submit Flatpak to Flathub
on:
workflow_call:
inputs:
version:
description: Semver string e.g. 2.0.1
required: true
type: string
tag:
description: Git tag e.g. v2.0.1
required: true
type: string
secrets:
FLATHUB_TOKEN:
required: false
workflow_dispatch:
inputs:
version:
description: Semver string e.g. 2.0.1
required: true
type: string
tag:
description: Git tag e.g. v2.0.1
required: true
type: string
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
submit-to-flathub:
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- name: Check if app is listed on Flathub
id: flathub_check
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
if gh repo view flathub/ai.elizaos.App >/dev/null 2>&1; then
echo "listed=true" >> "$GITHUB_OUTPUT"
else
echo "listed=false" >> "$GITHUB_OUTPUT"
echo "::warning::App not yet listed on Flathub. Submit the initial listing at https://github.com/flathub/flathub before using this workflow."
fi
- name: Checkout flatpak manifest
if: steps.flathub_check.outputs.listed == 'true'
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
ref: ${{ inputs.tag }}
submodules: false
- name: Update manifest version
if: steps.flathub_check.outputs.listed == 'true'
env:
GH_TOKEN: ${{ secrets.FLATHUB_TOKEN || secrets.GITHUB_TOKEN }}
run: |
VERSION="${{ inputs.version }}"
TAG="${{ inputs.tag }}"
echo "Preparing Flatpak update for ${VERSION}"
# Get the tarball sha256 for the release
TARBALL_URL="https://github.com/${{ github.repository }}/archive/refs/tags/${TAG}.tar.gz"
curl -L "$TARBALL_URL" -o /tmp/release.tar.gz
SHA256=$(sha256sum /tmp/release.tar.gz | awk '{print $1}')
echo "Release tarball SHA256: $SHA256"
echo "TARBALL_SHA256=$SHA256" >> "$GITHUB_ENV"
echo "TARBALL_URL=$TARBALL_URL" >> "$GITHUB_ENV"
- name: Create Flathub PR
if: steps.flathub_check.outputs.listed == 'true'
env:
GH_TOKEN: ${{ secrets.FLATHUB_TOKEN || secrets.GITHUB_TOKEN }}
run: |
VERSION="${{ inputs.version }}"
BRANCH="update-to-${VERSION}"
# Fork the Flathub repo if needed, create branch, push updated manifest, open PR
gh repo fork flathub/ai.elizaos.App --clone --remote
cd ai.elizaos.App
git checkout -b "$BRANCH"
# Update the manifest with the new version/sha256
# (Actual manifest update depends on the Flathub manifest format)
sed -i "s|tag: .*|tag: ${VERSION}|g" ai.elizaos.App.yml || true
sed -i "s|sha256: .*|sha256: ${TARBALL_SHA256}|g" ai.elizaos.App.yml || true
git config user.name "elizaOS CI"
git config user.email "ci@elizaos.ai"
git add -A
git commit -m "Update to $VERSION"
git push origin "$BRANCH"
gh pr create \
--repo flathub/ai.elizaos.App \
--title "Update to elizaOS App $VERSION" \
--body "Automated update to version $VERSION (tag: ${{ inputs.tag }})" \
--base master \
--head "$BRANCH"
+117
View File
@@ -0,0 +1,117 @@
name: gitleaks
# SOC2 CC7.1 — automated secret scanning on every PR and push to protected
# branches. Fails the run on any finding. Configuration: .gitleaks.toml at
# repo root.
on:
pull_request:
branches: ["main", "develop"]
push:
branches: ["main", "develop"]
# cancel superseded in-flight runs for the same PR to free hosted-runner
# capacity. only cancels pull_request runs; push runs on protected branches
# (main/develop) are never cancelled mid-flight.
# CI fan-out fix: the PR-scoped concurrency from #14069 fell back to
# github.run_id on push, so every develop push got a unique group and
# nothing ever superseded. A 37-merge wave queued thousands of push runs
# and starved the self-hosted fleet. Fall back to github.ref (all develop
# pushes share a group) and cancel superseded runs on push too. PR behavior
# is unchanged (pull_request.number still wins the key); merge_group and
# schedule events are not in the cancel set, so the merge queue and nightly
# runs always complete.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' || github.event_name == 'push' }}
permissions:
contents: read
jobs:
gitleaks:
name: gitleaks
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
steps:
- name: Checkout
# actions/checkout@v4
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
# Full history so gitleaks can scan the diff range on PRs.
fetch-depth: 0
- name: Install gitleaks (OSS binary — no license required)
# gitleaks/gitleaks-action@v2 requires a paid license for org repos.
# We download the OSS CLI directly so the scan remains free and
# honors our .gitleaks.toml allowlist + custom rules.
run: |
set -euo pipefail
GITLEAKS_VERSION=8.30.1
curl -sSL -o /tmp/gitleaks.tar.gz \
"https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz"
tar -xzf /tmp/gitleaks.tar.gz -C /tmp gitleaks
# Install without sudo so this works on any runner (github-hosted or
# self-hosted, which has no passwordless sudo): write to a user-writable
# dir and add it to PATH for the run step, not `sudo install /usr/local/bin`.
mkdir -p "$HOME/.local/bin"
install -m 0755 /tmp/gitleaks "$HOME/.local/bin/gitleaks"
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
"$HOME/.local/bin/gitleaks" version
- name: Run gitleaks
env:
EVENT_NAME: ${{ github.event_name }}
PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
BEFORE_SHA: ${{ github.event.before }}
CURRENT_SHA: ${{ github.sha }}
run: |
set -euo pipefail
RANGE_ARGS=()
if [[ "$EVENT_NAME" == "pull_request" ]]; then
RANGE_ARGS=(--log-opts "${PR_BASE_SHA}..${PR_HEAD_SHA}")
elif [[ -n "$BEFORE_SHA" && "$BEFORE_SHA" != "0000000000000000000000000000000000000000" ]]; then
RANGE_ARGS=(--log-opts "${BEFORE_SHA}..${CURRENT_SHA}")
elif ! git rev-parse -q --verify "${CURRENT_SHA}^" >/dev/null 2>&1; then
# Orphan/root tip commit: a squashed "clean baseline" release cut with
# no parent (the release process resets main/develop to a fresh root
# commit). `-1 <sha>` would diff it against the empty tree and re-flag
# every token-like string in the entire monorepo — generated docs
# dumps, benchmark fixtures, public chain addresses — the same
# full-history re-scan the tip-commit branch below exists to avoid, in
# its degenerate form. The baseline's content is a squash of develop
# commits each already scanned incrementally on their PR/push, so a
# full re-scan adds no coverage and only produces false-positive noise.
echo "gitleaks: tip ${CURRENT_SHA} is an orphan/root baseline commit (no parent);"
echo "its content was already scanned incrementally on develop. Skipping full-tree re-scan."
exit 0
else
# Push with no usable before-SHA (force-push, branch (re)creation, or
# an unknown range — common on main when it is reset/force-pushed).
# Scan only the tip commit, NOT the whole history: a full-history
# scan re-flags long-known historical artifacts (e.g. token-like
# strings inside vendored *.patch blobs) on every such push and costs
# ~8 min / 3 GB. New content still gets scanned via the PR range and
# this tip-commit scan; history is audited by the scheduled deep scan.
RANGE_ARGS=(--log-opts "-1 ${CURRENT_SHA}")
fi
gitleaks detect \
--config .gitleaks.toml \
--source . \
--verbose \
--redact \
--report-format sarif \
--report-path gitleaks.sarif \
--no-banner \
"${RANGE_ARGS[@]}"
- name: Upload SARIF
if: always()
# actions/upload-artifact@v7
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: gitleaks-sarif
path: gitleaks.sarif
if-no-files-found: ignore
retention-days: 7
+89
View File
@@ -0,0 +1,89 @@
# Nightly per-GPU voice-bench smoke run.
#
# STATUS: stub. Gated by workflow_dispatch input until a self-hosted CUDA GPU
# runner is wired. L4 (the cheapest GitHub-hosted GPU runner) is a reasonable
# starting target; for the full matrix (3090 / 4090 / 5090 / H200) we
# need a self-hosted pool.
#
# Per docs/inference/gpu-tier.md the per-GPU autotune profiles live in
# packages/inference/configs/gpu/, and `voice-bench bench gpu` prints
# the resolved plan for the host card. Once a real cuda PipelineDriver
# is wired (see packages/inference/voice-bench/README.md "Wiring the
# real pipeline"), this workflow runs the actual benchmark.
name: gpu-bench-nightly
on:
schedule:
# Nightly at 06:00 UTC. Adjust once we know what window the
# self-hosted runners are available.
- cron: "0 6 * * *"
workflow_dispatch:
inputs:
bundle:
description: "Bundle id (e.g. eliza-1-9b)"
required: false
default: "eliza-1-9b"
enable_gpu_bench:
description: "Run the scaffold GPU bench jobs"
required: false
type: boolean
default: false
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
detect:
name: detect host GPU
if: ${{ github.event_name == 'workflow_dispatch' && inputs.enable_gpu_bench == true }}
runs-on: [self-hosted, gpu-cuda-12.6]
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: canary
- name: install
run: bun install --frozen-lockfile || bun install --no-frozen-lockfile
- name: probe GPU + print autotune plan
run: |
bun run --cwd packages/inference/voice-bench bench gpu \
--bundle "${{ github.event.inputs.bundle || 'eliza-1-9b' }}" \
| tee gpu-plan.json
- name: upload plan
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: gpu-plan
path: gpu-plan.json
if-no-files-found: error
retention-days: 7
bench:
name: voice-bench (cuda)
needs: detect
if: ${{ github.event_name == 'workflow_dispatch' && inputs.enable_gpu_bench == true }}
runs-on: [self-hosted, gpu-cuda-12.6]
timeout-minutes: 60
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: canary
- name: install
run: bun install --frozen-lockfile || bun install --no-frozen-lockfile
- name: run voice-bench
run: |
bun run --cwd packages/inference/voice-bench bench \
--bundle "${{ github.event.inputs.bundle || 'eliza-1-9b' }}" \
--backend cuda \
--scenario all \
--runs 5 \
--output bench.json
- name: upload bench result
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: bench-${{ github.run_id }}
path: bench.json
if-no-files-found: error
retention-days: 7
@@ -0,0 +1,111 @@
name: Headscale API Key Health
# The Headscale admin key expires independently of deploy cadence. Probe the
# daemon-local API from the control-plane host so an expired or rebuild-lost key
# turns the nightly red without copying the admin credential into GitHub Actions.
on:
schedule:
- cron: "23 7 * * *"
workflow_dispatch:
inputs:
environment:
description: "Control-plane environment to probe"
type: choice
default: staging
options: [staging, production]
permissions:
contents: read
concurrency:
group: headscale-api-key-health-${{ github.event.inputs.environment || 'staging' }}
cancel-in-progress: false
jobs:
probe:
name: Probe ${{ github.event.inputs.environment || 'staging' }} Headscale key
runs-on: ubuntu-24.04
environment: ${{ github.event.inputs.environment || 'staging' }}
timeout-minutes: 5
env:
DEPLOY_HOST: ${{ secrets.ELIZA_PROVISIONING_HOST }}
DEPLOY_SSH_KEY: ${{ secrets.ELIZA_PROVISIONING_SSH_KEY }}
steps:
- name: Check probe configuration
run: |
fail=0
[ -n "$DEPLOY_HOST" ] || { echo "::error::missing secret ELIZA_PROVISIONING_HOST"; fail=1; }
[ -n "$DEPLOY_SSH_KEY" ] || { echo "::error::missing secret ELIZA_PROVISIONING_SSH_KEY"; fail=1; }
[ "$fail" = 0 ] || exit 1
- name: Probe daemon-local Headscale API
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2
with:
host: ${{ env.DEPLOY_HOST }}
username: deploy
key: ${{ env.DEPLOY_SSH_KEY }}
command_timeout: 2m
script: |
set -eu
ENV_FILE=/opt/eliza/cloud/.env.local
RESPONSE_FILE=$(mktemp)
trap 'rm -f "$RESPONSE_FILE"' EXIT
read_env_value() {
key="$1"
line=$(sudo grep -m1 "^${key}=" "$ENV_FILE" || true)
value="${line#*=}"
if [ "${value#\"}" != "$value" ] && [ "${value%\"}" != "$value" ]; then
value="${value#\"}"
value="${value%\"}"
fi
printf '%s' "$value"
}
sudo test -r "$ENV_FILE" || {
echo "::error::Headscale environment file is unreadable on the control-plane host"
exit 1
}
HEADSCALE_API_URL=$(read_env_value HEADSCALE_API_URL)
HEADSCALE_API_KEY=$(read_env_value HEADSCALE_API_KEY)
[ -n "$HEADSCALE_API_URL" ] || {
echo "::error::HEADSCALE_API_URL is missing from the control-plane environment"
exit 1
}
[ -n "$HEADSCALE_API_KEY" ] || {
echo "::error::HEADSCALE_API_KEY is missing from the control-plane environment"
exit 1
}
endpoint="${HEADSCALE_API_URL%/}/api/v1/user"
status=$(curl --silent --show-error --connect-timeout 5 --max-time 20 \
--output "$RESPONSE_FILE" --write-out '%{http_code}' \
--header "Authorization: Bearer $HEADSCALE_API_KEY" \
"$endpoint") || {
echo "::error::Headscale API probe could not reach the daemon-local endpoint"
exit 1
}
case "$status" in
200)
node -e '
const fs = require("node:fs");
const payload = JSON.parse(fs.readFileSync(process.argv[1], "utf8"));
if (!payload || !Array.isArray(payload.users)) process.exit(1);
' "$RESPONSE_FILE" || {
echo "::error::Headscale API returned HTTP 200 with an invalid user-list payload"
exit 1
}
echo "Headscale API key accepted by the daemon-local endpoint."
;;
401|403)
echo "::error::Headscale rejected the configured API key with HTTP $status; rotate it before provisioning fails"
exit 1
;;
*)
echo "::error::Headscale API key probe returned unexpected HTTP $status"
exit 1
;;
esac
+46
View File
@@ -0,0 +1,46 @@
name: Hetzner E2E Reaper
# Belt-and-suspenders cleanup. Runs every 30 minutes and deletes any
# `ci=true,workflow=hetzner-e2e` servers older than 60 minutes — in
# case a crashed workflow leaks one. Gracefully no-ops when
# HCLOUD_TOKEN_CI isn't set so it doesn't spam-fail before secrets
# are configured.
on:
schedule:
- cron: '*/30 * * * *'
workflow_dispatch:
concurrency:
group: hetzner-e2e-reaper
cancel-in-progress: false
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
reap:
name: Sweep stale CI servers
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
environment: ci-hetzner-e2e
timeout-minutes: 10
env:
HCLOUD_TOKEN_CI: ${{ secrets.HCLOUD_TOKEN_CI }}
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: canary
- name: Install dependencies
run: bun install --ignore-scripts
- name: Ensure generated shared i18n data
run: bun packages/app-core/scripts/ensure-shared-i18n-data.mjs
- name: Reap
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-reaper.ts
+428
View File
@@ -0,0 +1,428 @@
name: Hetzner Agent E2E
# Nightly end-to-end smoke that provisions a fresh Hetzner cpx11 server,
# waits for cloud-init, deploys a trivial agent via the Eliza Cloud
# staging API, runs a bridge-ping healthcheck plus one REAL chat turn
# (message.send through the production bridge path — #15603 A3), then
# tears everything down. Gracefully skips when the required secrets are
# unset (so this workflow can land on develop and be activated later), but a
# configured secret that is rejected by Hetzner or the Cloud API is a red
# workflow: green-by-dead-credential is indistinguishable from a real pass.
#
# Required secrets / config (GitHub environment: ci-hetzner-e2e):
# HCLOUD_TOKEN_CI - Hetzner Cloud API token (CI-scoped project)
# CLOUD_E2E_API_KEY - Eliza Cloud staging bearer token (long-lived)
# CI_SSH_PRIVATE_KEY - OpenSSH private key (matches the public key
# uploaded to Hetzner)
# CI_SSH_PUBLIC_KEY_ID - Numeric Hetzner SSH key id for the above
#
# See packages/scripts/cloud/admin/hetzner-e2e/README.md for setup.
on:
schedule:
- cron: '0 7 * * *' # 07:00 UTC nightly
workflow_dispatch:
push:
branches: [develop]
paths:
- '.github/workflows/hetzner-e2e.yml'
- '.github/workflows/hetzner-e2e-reaper.yml'
- 'packages/scripts/cloud/admin/hetzner-e2e/**'
- 'packages/cloud/shared/src/lib/services/containers/**'
- 'packages/scripts/cloud/admin/daemons/**'
- 'packages/scripts/cloud/admin/bootstrap-provisioning-worker-host.mjs'
concurrency:
group: hetzner-e2e
cancel-in-progress: false
# Default to least privilege. Override per-job where needed.
permissions:
contents: read
jobs:
deploy:
name: Provision + deploy + healthcheck
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
environment: ci-hetzner-e2e
timeout-minutes: 25
outputs:
provisioned: ${{ steps.provision.outputs.provisioned }}
env:
HCLOUD_TOKEN_CI: ${{ secrets.HCLOUD_TOKEN_CI }}
CLOUD_E2E_API_KEY: ${{ secrets.CLOUD_E2E_API_KEY || secrets.ELIZACLOUD_API_KEY }}
CI_SSH_PRIVATE_KEY: ${{ secrets.CI_SSH_PRIVATE_KEY }}
CI_SSH_PUBLIC_KEY_ID: ${{ secrets.CI_SSH_PUBLIC_KEY_ID }}
HETZNER_E2E_STATE_FILE: /tmp/hetzner-e2e-state.json
steps:
- name: Check secret configuration
id: secret_config
run: |
missing=()
[ -z "$HCLOUD_TOKEN_CI" ] && missing+=("HCLOUD_TOKEN_CI")
[ -z "$CLOUD_E2E_API_KEY" ] && missing+=("CLOUD_E2E_API_KEY")
[ -z "$CI_SSH_PRIVATE_KEY" ] && missing+=("CI_SSH_PRIVATE_KEY")
[ -z "$CI_SSH_PUBLIC_KEY_ID" ] && missing+=("CI_SSH_PUBLIC_KEY_ID")
if [ "${#missing[@]}" -ne 0 ]; then
echo "configured=false" >> "$GITHUB_OUTPUT"
echo "::warning::Missing secrets: ${missing[*]}; skipping Hetzner E2E."
{
echo "### Hetzner E2E skipped"
echo ""
echo "Missing secrets in GitHub environment \`ci-hetzner-e2e\`:"
for s in "${missing[@]}"; do echo "- \`$s\`"; done
echo ""
echo "See \`packages/scripts/cloud/admin/hetzner-e2e/README.md\` for setup."
} >> "$GITHUB_STEP_SUMMARY"
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
echo "::error::Manual dispatch requires all hetzner-e2e secrets."
exit 1
fi
exit 0
fi
echo "configured=true" >> "$GITHUB_OUTPUT"
- name: Checkout
if: steps.secret_config.outputs.configured == 'true'
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Bun
if: steps.secret_config.outputs.configured == 'true'
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: canary
- name: Debug workspace
if: steps.secret_config.outputs.configured == 'true'
run: |
echo "cwd=$(pwd)"
echo "workspace=$GITHUB_WORKSPACE"
ls -la "$GITHUB_WORKSPACE" | head -30
test -f "$GITHUB_WORKSPACE/package.json" && echo HAS_PKG || echo NO_PKG
- name: Install dependencies
if: steps.secret_config.outputs.configured == 'true'
run: bun install --no-save --ignore-scripts
- name: Ensure generated shared i18n data
if: steps.secret_config.outputs.configured == 'true'
run: bun packages/app-core/scripts/ensure-shared-i18n-data.mjs
- name: Check cloud API auth
id: cloud_auth
if: steps.secret_config.outputs.configured == 'true'
run: |
set +e
response="$(
bun -e '
const base = (process.env.CLOUD_SMOKE_BASE_URL ?? "https://api-staging.elizacloud.ai").replace(/\/+$/, "");
const response = await fetch(`${base}/api/v1/eliza/agents`, {
headers: {
authorization: `Bearer ${process.env.CLOUD_E2E_API_KEY ?? ""}`,
accept: "application/json",
"user-agent": "hetzner-e2e-preflight/1.0",
},
signal: AbortSignal.timeout(30_000),
});
const text = await response.text();
console.log(JSON.stringify({ status: response.status, body: text.slice(0, 500) }));
'
)"
status=$?
set -e
if [ "$status" -ne 0 ]; then
echo "ready=false" >> "$GITHUB_OUTPUT"
echo "::warning::Cloud API auth preflight could not reach staging; skipping Hetzner E2E."
{
echo "### Hetzner E2E skipped"
echo ""
echo "Cloud API auth preflight could not reach staging before provisioning Hetzner capacity."
echo ""
echo "\`\`\`json"
echo "$response"
echo "\`\`\`"
} >> "$GITHUB_STEP_SUMMARY"
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
exit 1
fi
exit 0
fi
status_code="$(printf '%s' "$response" | bun -e 'const chunks=[]; for await (const chunk of Bun.stdin.stream()) chunks.push(Buffer.from(chunk)); const data = JSON.parse(Buffer.concat(chunks).toString() || "{}"); console.log(data.status ?? "");')"
if [ "$status_code" = "401" ] || [ "$status_code" = "403" ]; then
echo "ready=false" >> "$GITHUB_OUTPUT"
echo "::error::Cloud API rejected CLOUD_E2E_API_KEY with HTTP $status_code; refresh the ci-hetzner-e2e environment secret."
{
echo "### Hetzner E2E blocked by rejected Cloud API credential"
echo ""
echo "Cloud API rejected \`CLOUD_E2E_API_KEY\` with HTTP \`$status_code\` before provisioning Hetzner capacity."
echo ""
echo "Refresh the \`CLOUD_E2E_API_KEY\` / \`ELIZACLOUD_API_KEY\` secret in the \`ci-hetzner-e2e\` environment, or check the staging API-key lookup path."
} >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
if [ "${status_code#2}" != "$status_code" ]; then
echo "ready=true" >> "$GITHUB_OUTPUT"
exit 0
fi
if [ "${status_code#5}" != "$status_code" ]; then
echo "ready=false" >> "$GITHUB_OUTPUT"
echo "::warning::Cloud API auth preflight returned HTTP $status_code; skipping Hetzner E2E before provisioning."
{
echo "### Hetzner E2E skipped"
echo ""
echo "Cloud API auth preflight returned HTTP \`$status_code\` before provisioning Hetzner capacity."
echo ""
echo "\`\`\`json"
echo "$response"
echo "\`\`\`"
echo ""
echo "Check staging Cloud API logs for \`user-agent: hetzner-e2e-preflight/1.0\` on \`/api/v1/eliza/agents\`."
} >> "$GITHUB_STEP_SUMMARY"
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
exit 1
fi
exit 0
fi
echo "::error::Unexpected cloud API auth preflight response: $response"
exit 1
- name: Check Hetzner API token
id: hcloud_auth
if: steps.secret_config.outputs.configured == 'true' && steps.cloud_auth.outputs.ready == 'true'
run: |
set +e
response="$(
bun -e '
const response = await fetch("https://api.hetzner.cloud/v1/ssh_keys?per_page=1", {
headers: {
authorization: `Bearer ${process.env.HCLOUD_TOKEN_CI ?? ""}`,
accept: "application/json",
"user-agent": "hetzner-e2e-preflight/1.0",
},
signal: AbortSignal.timeout(30_000),
});
const text = await response.text();
console.log(JSON.stringify({ status: response.status, body: text.slice(0, 500) }));
'
)"
status=$?
set -e
if [ "$status" -ne 0 ]; then
echo "ready=false" >> "$GITHUB_OUTPUT"
echo "::warning::Hetzner API token preflight could not reach api.hetzner.cloud; skipping Hetzner E2E."
{
echo "### Hetzner E2E skipped"
echo ""
echo "Hetzner API token preflight could not reach \`api.hetzner.cloud\` before provisioning capacity."
echo ""
echo "\`\`\`json"
echo "$response"
echo "\`\`\`"
} >> "$GITHUB_STEP_SUMMARY"
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
exit 1
fi
exit 0
fi
status_code="$(printf '%s' "$response" | bun -e 'const chunks=[]; for await (const chunk of Bun.stdin.stream()) chunks.push(Buffer.from(chunk)); const data = JSON.parse(Buffer.concat(chunks).toString() || "{}"); console.log(data.status ?? "");')"
if [ "$status_code" = "401" ] || [ "$status_code" = "403" ]; then
echo "ready=false" >> "$GITHUB_OUTPUT"
echo "::error::Hetzner rejected HCLOUD_TOKEN_CI with HTTP $status_code; refresh the ci-hetzner-e2e environment secret."
{
echo "### Hetzner E2E blocked by rejected Hetzner credential"
echo ""
echo "Hetzner rejected \`HCLOUD_TOKEN_CI\` with HTTP \`$status_code\` before provisioning capacity."
echo ""
echo "Refresh \`HCLOUD_TOKEN_CI\` in the \`ci-hetzner-e2e\` environment, or confirm the Hetzner project is still active."
} >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
if [ "${status_code#2}" != "$status_code" ]; then
echo "ready=true" >> "$GITHUB_OUTPUT"
exit 0
fi
if [ "${status_code#5}" != "$status_code" ]; then
echo "ready=false" >> "$GITHUB_OUTPUT"
echo "::warning::Hetzner API token preflight returned HTTP $status_code; skipping Hetzner E2E before provisioning."
{
echo "### Hetzner E2E skipped"
echo ""
echo "Hetzner API token preflight returned HTTP \`$status_code\` before provisioning capacity."
echo ""
echo "\`\`\`json"
echo "$response"
echo "\`\`\`"
} >> "$GITHUB_STEP_SUMMARY"
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
exit 1
fi
exit 0
fi
echo "::error::Unexpected Hetzner API token preflight response: $response"
exit 1
- name: Provision Hetzner server
id: provision
if: steps.secret_config.outputs.configured == 'true' && steps.cloud_auth.outputs.ready == 'true' && steps.hcloud_auth.outputs.ready == 'true'
run: |
log=/tmp/hetzner-e2e-provision.log
set +e
bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-provision.ts 2>&1 | tee "$log"
rc=${PIPESTATUS[0]}
set -e
if [ "$rc" -eq 0 ]; then
echo "provisioned=true" >> "$GITHUB_OUTPUT"
exit 0
fi
# A rejected / expired / read-only HCLOUD_TOKEN_CI (or a deactivated project)
# surfaces as HTTP 401/403 missing_token / "permission denied" when CREATING the
# server — the read-only token preflight above can pass while this write fails.
# That is an operator secret problem, not a code regression, but the
# token is configured and rejected. Fail red for every event so the
# nightly cannot look healthy while real-infra coverage is disabled.
if grep -qE "Hetzner rejected the token|missing_token|permission denied|HTTP 401|HTTP 403|status: 401|status: 403" "$log"; then
echo "provisioned=false" >> "$GITHUB_OUTPUT"
echo "::error::Hetzner rejected HCLOUD_TOKEN_CI (HTTP 401/403); refresh the token in the ci-hetzner-e2e environment."
{
echo "### Hetzner E2E blocked by rejected Hetzner credential"
echo ""
echo "Hetzner rejected \`HCLOUD_TOKEN_CI\` (HTTP 401/403 \`missing_token\` / permission denied) when creating the server."
echo ""
echo "The token may be read-only or expired for writes. Refresh \`HCLOUD_TOKEN_CI\` in the \`ci-hetzner-e2e\` environment with a read-write CI token, or confirm the project is active."
} >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
# Any other provision failure is real — surface it (red).
echo "provisioned=false" >> "$GITHUB_OUTPUT"
exit "$rc"
- name: Surface provision failure diagnostic
if: always() && steps.secret_config.outputs.configured == 'true' && steps.cloud_auth.outputs.ready == 'true' && steps.hcloud_auth.outputs.ready == 'true' && steps.provision.outcome == 'failure'
run: |
log=/tmp/hetzner-e2e-provision.log
{
echo "### Hetzner E2E provisioning failed"
echo ""
if [ -f "$log" ] && grep -qE "quota exhausted|server limit reached|limit_reached|resource_limit_exceeded|quota_exceeded" "$log"; then
echo "**Cause:** Hetzner project quota exhausted (server cap reached)."
echo ""
echo "Operator action:"
echo "1. Open https://console.hetzner.cloud/ and check the CI project for leaked servers (filter labels \`ci=true\`, \`workflow=hetzner-e2e\`)."
echo "2. Delete any leaked servers, or wait for the half-hourly reaper workflow to sweep them."
echo "3. Re-run this workflow."
echo ""
echo "If the project itself has a tighter cap than the fallback ladder can survive, request a quota increase from Hetzner support or rotate \`HCLOUD_TOKEN_CI\` to a project with capacity."
elif [ -f "$log" ] && grep -qE "Hetzner rejected the token|missing_token|HTTP 401|HTTP 403" "$log"; then
echo "**Cause:** Hetzner rejected the API token (HTTP 401/403)."
echo ""
echo "Operator action: refresh \`HCLOUD_TOKEN_CI\` in the \`ci-hetzner-e2e\` GitHub environment, or confirm the project is still active."
else
echo "**Cause:** see the \"Provision Hetzner server\" step log for the raw error."
echo ""
echo "If the error message is unfamiliar, check \`packages/scripts/cloud/admin/hetzner-e2e/README.md\` and the fallback ladder in \`hetzner-e2e-provision.ts\`."
fi
} >> "$GITHUB_STEP_SUMMARY"
- name: Wait for host ready
if: steps.provision.outputs.provisioned == 'true'
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-wait-ready.ts
- name: Deploy trivial agent
if: steps.provision.outputs.provisioned == 'true'
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-deploy-agent.ts
- name: Healthcheck
if: steps.provision.outputs.provisioned == 'true'
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-healthcheck.ts
# A real user-message round-trip through the same Worker bridge path a
# production chat takes. status.get alone lets "provisioned but chat
# dead-ends" regressions (#15347) pass the nightly. Failure here fails
# the deploy job; the teardown job runs regardless (needs+always()).
- name: Chat turn
if: steps.provision.outputs.provisioned == 'true'
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-chat.ts
- name: Upload state artifact
if: always() && steps.provision.outputs.provisioned == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
with:
name: hetzner-e2e-state
path: /tmp/hetzner-e2e-state.json
if-no-files-found: ignore
retention-days: 7
teardown:
name: Teardown
needs: deploy
if: always()
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
environment: ci-hetzner-e2e
timeout-minutes: 10
env:
HCLOUD_TOKEN_CI: ${{ secrets.HCLOUD_TOKEN_CI }}
HETZNER_E2E_STATE_FILE: /tmp/hetzner-e2e-state.json
steps:
- name: Check secret configuration
id: secret_config
run: |
if [ -z "$HCLOUD_TOKEN_CI" ]; then
echo "configured=false" >> "$GITHUB_OUTPUT"
echo "::warning::HCLOUD_TOKEN_CI missing; nothing to tear down."
exit 0
fi
echo "configured=true" >> "$GITHUB_OUTPUT"
- name: Checkout
if: steps.secret_config.outputs.configured == 'true'
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Bun
if: steps.secret_config.outputs.configured == 'true'
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: canary
- name: Install dependencies
if: steps.secret_config.outputs.configured == 'true'
run: bun install --no-save --ignore-scripts
- name: Ensure generated shared i18n data
if: steps.secret_config.outputs.configured == 'true'
run: bun packages/app-core/scripts/ensure-shared-i18n-data.mjs
- name: Download state artifact
if: steps.secret_config.outputs.configured == 'true'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c
with:
name: hetzner-e2e-state
path: /tmp/hetzner-e2e-state
continue-on-error: true
- name: Restore state file
if: steps.secret_config.outputs.configured == 'true'
run: |
if [ -f /tmp/hetzner-e2e-state/hetzner-e2e-state.json ]; then
cp /tmp/hetzner-e2e-state/hetzner-e2e-state.json /tmp/hetzner-e2e-state.json
echo "State file restored:"
cat /tmp/hetzner-e2e-state.json
else
echo "No state artifact; teardown will fall back to label sweep."
fi
- name: Teardown
if: steps.secret_config.outputs.configured == 'true'
env:
# Lets the teardown script distinguish "dead token, nothing was
# provisioned, nothing can leak" (soft-skip) from "dead token but a
# server exists" (hard fail so the leak is never silent).
DEPLOY_PROVISIONED: ${{ needs.deploy.outputs.provisioned }}
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-teardown.ts
@@ -0,0 +1,81 @@
name: Hetzner Provision Smoke
# Manual real-Hetzner provision smoke. Provisions a fresh cx22 server, waits
# for cloud-init / SSH readiness, then tears it down — all in one job so the
# /tmp state file persists across steps. Deliberately bypasses the Eliza Cloud
# staging API (no auth preflight, no trivial-agent deploy) so the core Hetzner
# provisioning path can be authentically confirmed even when the staging Cloud
# API or its Redis is degraded. Uses the same `ci-hetzner-e2e` environment
# secrets and the same provision/wait/teardown scripts as the nightly e2e.
on:
workflow_dispatch:
concurrency:
group: hetzner-e2e
cancel-in-progress: false
permissions:
contents: read
jobs:
provision:
name: Provision + wait-ready + teardown (real Hetzner)
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
environment: ci-hetzner-e2e
timeout-minutes: 20
env:
HCLOUD_TOKEN_CI: ${{ secrets.HCLOUD_TOKEN_CI }}
CI_SSH_PRIVATE_KEY: ${{ secrets.CI_SSH_PRIVATE_KEY }}
CI_SSH_PUBLIC_KEY_ID: ${{ secrets.CI_SSH_PUBLIC_KEY_ID }}
HETZNER_E2E_STATE_FILE: /tmp/hetzner-e2e-state.json
# cx22 is deprecated and cax (ARM) isn't placeable for this project;
# cpx22 (x86 2c/4g) is available across the EU datacenters.
HETZNER_E2E_SERVER_TYPE: cpx22
HETZNER_E2E_LOCATION: nbg1
steps:
- name: Check secret configuration
run: |
missing=()
[ -z "$HCLOUD_TOKEN_CI" ] && missing+=("HCLOUD_TOKEN_CI")
[ -z "$CI_SSH_PRIVATE_KEY" ] && missing+=("CI_SSH_PRIVATE_KEY")
[ -z "$CI_SSH_PUBLIC_KEY_ID" ] && missing+=("CI_SSH_PUBLIC_KEY_ID")
if [ ${#missing[@]} -ne 0 ]; then
echo "::error::Missing ci-hetzner-e2e secrets: ${missing[*]}"
exit 1
fi
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Bun
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: canary
- name: Install dependencies
run: bun install --no-save --ignore-scripts
- name: Ensure generated shared i18n data
run: node packages/app-core/scripts/ensure-shared-i18n-data.mjs
- name: List Hetzner catalog (what's actually placeable)
run: |
echo "=== non-deprecated server types (id name arch cores/mem) ==="
curl -s -H "Authorization: Bearer $HCLOUD_TOKEN_CI" \
https://api.hetzner.cloud/v1/server_types \
| jq -r '.server_types[] | select(.deprecation==null) | "\(.id)\t\(.name)\t\(.architecture)\t\(.cores)c/\(.memory)g"' | sort -n
echo "=== per-datacenter available server-type IDs ==="
curl -s -H "Authorization: Bearer $HCLOUD_TOKEN_CI" \
https://api.hetzner.cloud/v1/datacenters \
| jq -r '.datacenters[] | "\(.name)\tavail=[\(.server_types.available | join(","))]"'
- name: Provision Hetzner server
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-provision.ts
- name: Wait for host ready (cloud-init + SSH)
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-wait-ready.ts
- name: Teardown (always)
if: always()
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-teardown.ts
@@ -0,0 +1,281 @@
name: Hetzner Sleep-Wake Smoke
# Real-infrastructure proof of the sleep -> back up -> de-provision the Hetzner
# box -> wake -> re-provision from backup -> restore lifecycle. Uses real
# Hetzner boxes via HCLOUD_TOKEN_CI + SSH; independent of the Eliza Cloud
# staging API (which is currently degraded). The backup transport here is the
# runner filesystem standing in for S3/R2 (CI has HCLOUD + SSH, not R2) — the
# point is to authentically exercise the provision -> capture state ->
# DESTROY THE BOX -> provision a NEW box -> restore -> verify cycle end to end.
#
# Secret gating mirrors hetzner-e2e.yml: missing HCLOUD_TOKEN_CI / SSH secrets
# skip cleanly (exit 0 + loud warning and step summary) so forks and inactive
# environments stay quiet. A configured credential that Hetzner rejects is red
# for every event: otherwise the real-infra nightly can look healthy while it
# never exercised anything.
#
# Required secrets (GitHub environment: ci-hetzner-e2e):
# HCLOUD_TOKEN_CI - Hetzner Cloud API token (CI-scoped project)
# CI_SSH_PRIVATE_KEY - OpenSSH private key (matches the public key
# uploaded to Hetzner)
# CI_SSH_PUBLIC_KEY_ID - Numeric Hetzner SSH key id for the above
on:
schedule:
# 06:00 UTC nightly — an hour before hetzner-e2e's 07:00 slot. Both
# workflows share the `hetzner-e2e` concurrency group (no cancel), so a
# long sleep-wake run queues the agent e2e instead of racing it for the
# CI project's server quota.
- cron: '0 6 * * *'
workflow_dispatch:
push:
branches: [claude/hetzner-provision-smoke]
concurrency:
group: hetzner-e2e
cancel-in-progress: false
permissions:
contents: read
jobs:
sleep-wake:
name: provision -> sleep(backup+destroy) -> wake(reprovision+restore)
runs-on: ${{ fromJSON(vars.HETZNER_FLEET_ONLINE == 'false' && '["ubuntu-24.04"]' || '["self-hosted","hetzner-robot"]') }}
environment: ci-hetzner-e2e
timeout-minutes: 30
env:
HCLOUD_TOKEN_CI: ${{ secrets.HCLOUD_TOKEN_CI }}
CI_SSH_PRIVATE_KEY: ${{ secrets.CI_SSH_PRIVATE_KEY }}
CI_SSH_PUBLIC_KEY_ID: ${{ secrets.CI_SSH_PUBLIC_KEY_ID }}
HETZNER_E2E_STATE_FILE: /tmp/hetzner-e2e-state.json
HETZNER_E2E_SERVER_TYPE: cpx22
HETZNER_E2E_LOCATION: nbg1
SSH_OPTS: -i /tmp/ci_ssh_key -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=15 -o BatchMode=yes
steps:
- name: Check secret configuration
id: secret_config
run: |
missing=()
[ -z "$HCLOUD_TOKEN_CI" ] && missing+=("HCLOUD_TOKEN_CI")
[ -z "$CI_SSH_PRIVATE_KEY" ] && missing+=("CI_SSH_PRIVATE_KEY")
[ -z "$CI_SSH_PUBLIC_KEY_ID" ] && missing+=("CI_SSH_PUBLIC_KEY_ID")
if [ "${#missing[@]}" -ne 0 ]; then
echo "configured=false" >> "$GITHUB_OUTPUT"
echo "::warning::Missing secrets: ${missing[*]}; skipping Hetzner sleep-wake smoke."
{
echo "### Hetzner sleep-wake smoke skipped"
echo ""
echo "Missing secrets in GitHub environment \`ci-hetzner-e2e\`:"
for s in "${missing[@]}"; do echo "- \`$s\`"; done
echo ""
echo "See \`packages/scripts/cloud/admin/hetzner-e2e/README.md\` for setup."
} >> "$GITHUB_STEP_SUMMARY"
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
echo "::error::Manual dispatch requires all hetzner-e2e secrets."
exit 1
fi
exit 0
fi
echo "configured=true" >> "$GITHUB_OUTPUT"
- name: Checkout
if: steps.secret_config.outputs.configured == 'true'
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
- name: Setup Bun
if: steps.secret_config.outputs.configured == 'true'
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6
with:
bun-version: canary
- name: Install deps
if: steps.secret_config.outputs.configured == 'true'
run: bun install --no-save --ignore-scripts
- name: Ensure generated shared i18n data
if: steps.secret_config.outputs.configured == 'true'
run: node packages/app-core/scripts/ensure-shared-i18n-data.mjs
- name: Check Hetzner API token
id: hcloud_auth
if: steps.secret_config.outputs.configured == 'true'
run: |
set +e
response="$(
bun -e '
const response = await fetch("https://api.hetzner.cloud/v1/ssh_keys?per_page=1", {
headers: {
authorization: `Bearer ${process.env.HCLOUD_TOKEN_CI ?? ""}`,
accept: "application/json",
"user-agent": "hetzner-sleep-wake-preflight/1.0",
},
signal: AbortSignal.timeout(30_000),
});
const text = await response.text();
console.log(JSON.stringify({ status: response.status, body: text.slice(0, 500) }));
'
)"
status=$?
set -e
if [ "$status" -ne 0 ]; then
echo "ready=false" >> "$GITHUB_OUTPUT"
echo "::warning::Hetzner API token preflight could not reach api.hetzner.cloud; skipping sleep-wake smoke."
{
echo "### Hetzner sleep-wake smoke skipped"
echo ""
echo "Hetzner API token preflight could not reach \`api.hetzner.cloud\` before provisioning capacity."
echo ""
echo "\`\`\`json"
echo "$response"
echo "\`\`\`"
} >> "$GITHUB_STEP_SUMMARY"
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
exit 1
fi
exit 0
fi
status_code="$(printf '%s' "$response" | bun -e 'const chunks=[]; for await (const chunk of Bun.stdin.stream()) chunks.push(Buffer.from(chunk)); const data = JSON.parse(Buffer.concat(chunks).toString() || "{}"); console.log(data.status ?? "");')"
if [ "$status_code" = "401" ] || [ "$status_code" = "403" ]; then
echo "ready=false" >> "$GITHUB_OUTPUT"
echo "::error::Hetzner rejected HCLOUD_TOKEN_CI with HTTP $status_code; refresh the ci-hetzner-e2e environment secret."
{
echo "### Hetzner sleep-wake smoke blocked by rejected Hetzner credential"
echo ""
echo "Hetzner rejected \`HCLOUD_TOKEN_CI\` with HTTP \`$status_code\` before provisioning capacity."
echo ""
echo "Refresh \`HCLOUD_TOKEN_CI\` in the \`ci-hetzner-e2e\` environment, or confirm the Hetzner project is still active."
} >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
if [ "${status_code#2}" != "$status_code" ]; then
echo "ready=true" >> "$GITHUB_OUTPUT"
exit 0
fi
if [ "${status_code#5}" != "$status_code" ]; then
echo "ready=false" >> "$GITHUB_OUTPUT"
echo "::warning::Hetzner API token preflight returned HTTP $status_code; skipping sleep-wake smoke before provisioning."
{
echo "### Hetzner sleep-wake smoke skipped"
echo ""
echo "Hetzner API token preflight returned HTTP \`$status_code\` before provisioning capacity."
echo ""
echo "\`\`\`json"
echo "$response"
echo "\`\`\`"
} >> "$GITHUB_STEP_SUMMARY"
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
exit 1
fi
exit 0
fi
echo "::error::Unexpected Hetzner API token preflight response: $response"
exit 1
- name: Write SSH key
if: steps.secret_config.outputs.configured == 'true' && steps.hcloud_auth.outputs.ready == 'true'
run: |
printf '%s\n' "$CI_SSH_PRIVATE_KEY" > /tmp/ci_ssh_key
chmod 600 /tmp/ci_ssh_key
# ── Provision box A + boot ──────────────────────────────────────────
- name: Provision box A
id: provision_a
if: steps.secret_config.outputs.configured == 'true' && steps.hcloud_auth.outputs.ready == 'true'
run: |
log=/tmp/hetzner-sleep-wake-provision.log
set +e
bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-provision.ts 2>&1 | tee "$log"
rc=${PIPESTATUS[0]}
set -e
if [ "$rc" -eq 0 ]; then
echo "provisioned=true" >> "$GITHUB_OUTPUT"
exit 0
fi
# A rejected / expired / read-only HCLOUD_TOKEN_CI (or a deactivated
# project) surfaces as HTTP 401/403 when CREATING the server — the
# read-only preflight above can pass while this write fails. That is
# an operator secret problem, not a code regression, but it must fail
# red so a scheduled run cannot look green while the token is dead.
if grep -qE "Hetzner rejected the token|missing_token|permission denied|HTTP 401|HTTP 403|status: 401|status: 403" "$log"; then
echo "provisioned=false" >> "$GITHUB_OUTPUT"
echo "::error::Hetzner rejected HCLOUD_TOKEN_CI (HTTP 401/403); refresh the token in the ci-hetzner-e2e environment."
{
echo "### Hetzner sleep-wake smoke blocked by rejected Hetzner credential"
echo ""
echo "Hetzner rejected \`HCLOUD_TOKEN_CI\` (HTTP 401/403 \`missing_token\` / permission denied) when creating the server."
echo ""
echo "The token may be read-only or expired for writes. Refresh \`HCLOUD_TOKEN_CI\` in the \`ci-hetzner-e2e\` environment with a read-write CI token, or confirm the project is active."
} >> "$GITHUB_STEP_SUMMARY"
exit 1
fi
# Any other provision failure is real — surface it (red).
echo "provisioned=false" >> "$GITHUB_OUTPUT"
exit "$rc"
- name: Wait A ready
if: steps.provision_a.outputs.provisioned == 'true'
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-wait-ready.ts
# ── Write agent state on A, then BACK IT UP off-box ─────────────────
- name: Write + back up agent state from A
if: steps.provision_a.outputs.provisioned == 'true'
run: |
IP_A=$(jq -r .ip "$HETZNER_E2E_STATE_FILE"); ID_A=$(jq -r .id "$HETZNER_E2E_STATE_FILE")
echo "box A: id=$ID_A ip=$IP_A"
echo "$ID_A" > /tmp/box-a-id
# Simulate agent state (mirrors AgentBackupStateData shape).
STATE='{"memories":[{"role":"user","text":"remember: orbital-mango-4417","timestamp":1},{"role":"assistant","text":"noted","timestamp":2}],"config":{"name":"sleepwake-probe"},"workspaceFiles":{"note.md":"survive the sleep"}}'
ssh $SSH_OPTS root@"$IP_A" "mkdir -p /root/agent && cat > /root/agent/state.json" <<< "$STATE"
# "Snapshot to S3" — here the runner FS stands in for R2.
scp $SSH_OPTS root@"$IP_A":/root/agent/state.json /tmp/backup-A.json
sha256sum /tmp/backup-A.json | awk '{print $1}' > /tmp/backup-A.sha
echo "backup sha: $(cat /tmp/backup-A.sha)"
test -s /tmp/backup-A.json
# ── SLEEP: de-provision (destroy) box A ─────────────────────────────
- name: Sleep — de-provision (destroy) box A
if: steps.provision_a.outputs.provisioned == 'true'
run: |
bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-teardown.ts
ID_A=$(cat /tmp/box-a-id)
code=$(curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $HCLOUD_TOKEN_CI" "https://api.hetzner.cloud/v1/servers/$ID_A")
echo "GET server $ID_A after teardown -> $code (expect 404)"
[ "$code" = "404" ] || { echo "::error::box A not destroyed"; exit 1; }
# ── WAKE: re-provision a NEW box B + restore from backup ────────────
- name: Wake — re-provision box B
if: steps.provision_a.outputs.provisioned == 'true'
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-provision.ts
- name: Wait B ready
if: steps.provision_a.outputs.provisioned == 'true'
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-wait-ready.ts
- name: Restore backup onto B + verify state survived
if: steps.provision_a.outputs.provisioned == 'true'
run: |
IP_B=$(jq -r .ip "$HETZNER_E2E_STATE_FILE"); ID_B=$(jq -r .id "$HETZNER_E2E_STATE_FILE")
ID_A=$(cat /tmp/box-a-id)
echo "box B: id=$ID_B ip=$IP_B (was A id=$ID_A)"
[ "$ID_B" != "$ID_A" ] || { echo "::error::wake did not create a NEW box"; exit 1; }
scp $SSH_OPTS /tmp/backup-A.json root@"$IP_B":/root/agent-state-restored.json
REMOTE_SHA=$(ssh $SSH_OPTS root@"$IP_B" "sha256sum /root/agent-state-restored.json | awk '{print \$1}'")
echo "restored sha on B: $REMOTE_SHA ; backup sha: $(cat /tmp/backup-A.sha)"
[ "$REMOTE_SHA" = "$(cat /tmp/backup-A.sha)" ] || { echo "::error::restored state mismatch"; exit 1; }
ssh $SSH_OPTS root@"$IP_B" "grep -q orbital-mango-4417 /root/agent-state-restored.json" \
&& echo "PASS: agent state survived sleep(destroy A) -> wake(new box B) -> restore"
# Runs whenever provisioning was ATTEMPTED (success or failure — a failed
# create can still leak a server), and skips only when provisioning never
# started. DEPLOY_PROVISIONED lets the teardown script distinguish "dead
# token, nothing was provisioned, nothing can leak" (soft-skip) from
# "dead token but a server exists" (hard fail so the leak is never
# silent) — same contract as hetzner-e2e.yml's teardown job.
- name: Teardown box B (always)
if: always() && steps.provision_a.outcome != 'skipped'
env:
DEPLOY_PROVISIONED: ${{ steps.provision_a.outputs.provisioned }}
run: bun run packages/scripts/cloud/admin/hetzner-e2e/hetzner-e2e-teardown.ts

Some files were not shown because too many files have changed in this diff Show More