chore: import upstream snapshot with attribution
Deploy Site / deploy-vercel (push) Has been skipped
Deploy Site / deploy-docs (push) Has been skipped
Build Skills Index / build-index (push) Has been skipped
CI / Deny unrelated histories (push) Has been skipped
CI / Detect affected areas (push) Successful in 27m35s
CI / OSV scan (push) Failing after 4s
CI / Build&Test Docker image (push) Successful in 9s
CI / Supply-chain scan (push) Has been skipped
CI / Lint Docker scripts (push) Failing after 5m13s
CI / Check contributors (push) Failing after 12m8s
CI / Docs Site (push) Failing after 12m8s
CI / TypeScript (push) Failing after 12m8s
CI / Python lints (push) Failing after 12m9s
CI / Python tests (push) Failing after 12m9s
CI / Check uv.lock (push) Failing after 23m22s
CI / CI timing report (push) Has been cancelled
Build Skills Index / trigger-deploy (push) Has been cancelled
CI / All required checks pass (push) Has been cancelled
Deploy Site / deploy-vercel (push) Has been skipped
Deploy Site / deploy-docs (push) Has been skipped
Build Skills Index / build-index (push) Has been skipped
CI / Deny unrelated histories (push) Has been skipped
CI / Detect affected areas (push) Successful in 27m35s
CI / OSV scan (push) Failing after 4s
CI / Build&Test Docker image (push) Successful in 9s
CI / Supply-chain scan (push) Has been skipped
CI / Lint Docker scripts (push) Failing after 5m13s
CI / Check contributors (push) Failing after 12m8s
CI / Docs Site (push) Failing after 12m8s
CI / TypeScript (push) Failing after 12m8s
CI / Python lints (push) Failing after 12m9s
CI / Python tests (push) Failing after 12m9s
CI / Check uv.lock (push) Failing after 23m22s
CI / CI timing report (push) Has been cancelled
Build Skills Index / trigger-deploy (push) Has been cancelled
CI / All required checks pass (push) Has been cancelled
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,208 @@
|
||||
# Chronos managed-cron — agent ↔ NAS wire contract
|
||||
|
||||
**Status:** authoritative wire spec for the Chronos cron provider.
|
||||
**Audience:** the NAS-side implementer of the `agent-cron` endpoints
|
||||
(`nous-account-service`) and anyone debugging the managed-cron path.
|
||||
|
||||
Chronos lets a hosted Hermes gateway **scale to zero** while idle and still
|
||||
fire cron jobs. Instead of an in-process 60-second ticker, the agent asks NAS
|
||||
to arm exactly **one external one-shot per job at that job's real next-fire
|
||||
time**. NAS calls the agent back at fire time over an authenticated webhook;
|
||||
the agent runs the job and re-arms the next one-shot. Between fires the agent
|
||||
process can be fully stopped — it wakes only on a genuine fire.
|
||||
|
||||
The external scheduler NAS uses to implement the one-shots is an **internal NAS
|
||||
implementation detail**. The agent never talks to it, never holds its
|
||||
credentials, and never names it. The agent only knows the three NAS endpoints
|
||||
below.
|
||||
|
||||
```
|
||||
create/update/pause/resume/remove a cron job (agent side)
|
||||
│
|
||||
▼
|
||||
ChronosCronScheduler.reconcile() ── agent computes next_run_at
|
||||
│ POST {portal}/api/agent-cron/provision (auth: agent's Nous access token)
|
||||
▼
|
||||
NAS arms a one-shot for fire_at ── NAS owns the scheduler + its creds
|
||||
│
|
||||
⏰ at fire_at
|
||||
▼
|
||||
scheduler → POST {portal}/api/agent-cron/relay (auth: scheduler signature, NAS-verified)
|
||||
│
|
||||
▼
|
||||
NAS mints a short-lived agent-audience JWT (purpose=cron_fire)
|
||||
│ POST {agent_callback_url}/api/cron/fire (auth: that JWT)
|
||||
▼
|
||||
agent verifies the NAS JWT → store CAS claim → run_one_job → re-arm next one-shot
|
||||
```
|
||||
|
||||
## Trust model (read this first)
|
||||
|
||||
| Hop | Who calls whom | Auth mechanism | Verified by |
|
||||
|---|---|---|---|
|
||||
| 1 | agent → NAS (`provision`/`cancel`/`list`) | the agent's existing **Nous Portal access token** (Bearer) — for a hosted agent this is the **bootstrap-session token** NAS planted in `auth.json` (client `hermes-cli-vps`), NOT an `agent:*` client token | NAS (its normal agent-token path) |
|
||||
| 2 | scheduler → NAS (`relay`) | the scheduler's request **signature** | NAS (the signature path it already has) |
|
||||
| 3 | NAS → agent (`/api/cron/fire`) | a **short-lived NAS-minted JWT** (`aud=agent:{instance_id}`, `purpose=cron_fire`) | agent (PyJWT against NAS JWKS) |
|
||||
|
||||
> **Which token, exactly (hop 1).** A hosted agent never holds an `agent:{instance_id}`
|
||||
> OAuth client credential — that shape is minted only by the interactive dashboard
|
||||
> auth-code grant (a browser user). For all of its own outbound portal calls the
|
||||
> agent uses the **bootstrap-session access token** (`resolve_nous_access_token`),
|
||||
> minted under the bootstrap-only client `hermes-cli-vps` and seeded into the
|
||||
> container on first boot. NAS therefore must resolve the calling agent's instance
|
||||
> id from EITHER an `agent:{id}` client (self-hosted/dashboard callers) OR — for the
|
||||
> bootstrap token — from `AgentInstance.bootstrapSessionId` matching the token's
|
||||
> session id (`sid`), org-scoped. The fire JWT minted at hop 3 still carries
|
||||
> `aud=agent:{instance_id}` regardless. (Gating hop 1 on an `agent:*` client alone
|
||||
> 403s every real hosted-agent provision — see `src/server/agent-cron/instance-auth.ts`.)
|
||||
|
||||
Why NAS-mediated rather than scheduler→agent direct: the scheduler signs with
|
||||
**NAS's** keys, which the agent does not (and should not) hold. The agent can
|
||||
only verify a **NAS-minted** token — a trust path it already has. This keeps
|
||||
all scheduler credentials inside NAS. (Full rationale: the plan's DQ-4.)
|
||||
|
||||
No new secret is introduced on the agent: hop 1 reuses the token the agent
|
||||
already uses for the portal, and hop 3 reuses the NAS-JWT verification the agent
|
||||
already performs.
|
||||
|
||||
---
|
||||
|
||||
## Endpoint 1 — `POST /api/agent-cron/provision` (agent → NAS)
|
||||
|
||||
Arm (or re-arm, idempotently) exactly one one-shot for a job.
|
||||
|
||||
- **Auth:** `Authorization: Bearer <agent Nous access token>`. NAS validates via
|
||||
its normal agent-token path and scopes the row to the calling agent/org.
|
||||
- **Request body:**
|
||||
```json
|
||||
{
|
||||
"job_id": "ab12cd34",
|
||||
"fire_at": "2026-06-18T12:34:56+00:00",
|
||||
"agent_callback_url": "https://agent-xyz.fly.dev",
|
||||
"dedup_key": "ab12cd34:2026-06-18T12:34:56+00:00"
|
||||
}
|
||||
```
|
||||
- `fire_at` — ISO 8601, **agent-computed**. May be sub-minute in the future;
|
||||
NAS must honor second-granularity (the agent owns the time, so there is no
|
||||
1-minute scheduler floor).
|
||||
- `agent_callback_url` — the agent's own publicly-reachable base URL. NAS
|
||||
POSTs `{agent_callback_url}/api/cron/fire` at fire time.
|
||||
- `dedup_key` — `"{job_id}:{fire_at}"`. NAS **upserts by `(agent_id, job_id)`**
|
||||
so re-arming the same fire is idempotent (no duplicate one-shots). A new
|
||||
`fire_at` for the same `job_id` replaces the prior arm.
|
||||
- **Action:** arm one one-shot to fire at `fire_at`, destined for the NAS
|
||||
**relay** route (Endpoint 3) — NOT the agent directly, so NAS stays in the
|
||||
loop to mint the agent JWT. Persist `(agent_id, job_id, schedule_id,
|
||||
agent_callback_url)`.
|
||||
- **Response:** `200 {"schedule_id": "<opaque>"}`.
|
||||
|
||||
## Endpoint 2 — `POST /api/agent-cron/cancel` (agent → NAS)
|
||||
|
||||
- **Auth:** same as Endpoint 1.
|
||||
- **Body:** `{"job_id": "ab12cd34"}`.
|
||||
- **Action:** cancel the armed one-shot for `(agent_id, job_id)` and delete the
|
||||
row. Idempotent — cancelling an unknown job is a 200 no-op.
|
||||
- **Response:** `200 {"ok": true}`.
|
||||
|
||||
## Endpoint 3 — `POST /api/agent-cron/relay` (scheduler → NAS, the fire relay)
|
||||
|
||||
- **Auth:** the scheduler's request **signature**, verified by NAS with the
|
||||
signature path it already has. This is the trust boundary for the fire — a
|
||||
forged relay call must be rejected here.
|
||||
- **Action:**
|
||||
1. Look up `(agent_id, job_id) → agent_callback_url` from the persisted row.
|
||||
2. Mint a **short-lived** JWT: `aud = "agent:{instance_id}"`,
|
||||
`iss = {portal_url}`, `purpose = "cron_fire"`, small `exp` (≈60–120s),
|
||||
signed with NAS's normal asymmetric signing key (published via JWKS).
|
||||
3. `POST {agent_callback_url}/api/cron/fire` with
|
||||
`Authorization: Bearer <that JWT>` and body `{"job_id": "...", "fire_at": "..."}`.
|
||||
4. Treat a non-2xx agent response as a **retryable** failure (let the
|
||||
scheduler retry the relay). The agent's store CAS de-dupes a double fire,
|
||||
so retries are safe.
|
||||
- **Response to the scheduler:** 2xx once the agent POST is accepted (202), so
|
||||
the scheduler does not retry a delivered fire.
|
||||
|
||||
---
|
||||
|
||||
## Inbound `POST /api/cron/fire` (NAS → agent) — agent side, already implemented
|
||||
|
||||
This is the agent endpoint NAS calls in Endpoint 3 step 3. Served by the
|
||||
**dashboard app** (`hermes_cli/web_server.py`) — the agent's always-reachable
|
||||
public HTTP surface on hosted deployments (the gateway may be idle/scaled down);
|
||||
it is in `PUBLIC_API_PATHS` so the dashboard cookie gate lets the bearer-JWT
|
||||
callback through to the verifier. (Also registered on the optional
|
||||
`APIServerAdapter` for self-host API-server deployments.) The verifier is
|
||||
`plugins/cron/chronos/verify.py`.
|
||||
|
||||
- **Auth:** `Authorization: Bearer <NAS-minted JWT>`. The agent verifies:
|
||||
- signature against the NAS JWKS (`cron.chronos.nas_jwks_url`),
|
||||
- `aud` == `cron.chronos.expected_audience` (this agent's
|
||||
`agent:{instance_id}`),
|
||||
- `iss` == `cron.chronos.portal_url`,
|
||||
- `exp` / `nbf` (30s leeway),
|
||||
- `purpose == "cron_fire"` — a general agent JWT (no/other purpose) is
|
||||
rejected so it can't be replayed against this endpoint.
|
||||
- **Body:** `{"job_id": "ab12cd34", "fire_at": "..."}` (only `job_id` is used).
|
||||
- **Behavior:**
|
||||
- invalid/missing/forged/expired/wrong-aud/wrong-purpose token → **401**, no
|
||||
execution.
|
||||
- missing `job_id` → **400**.
|
||||
- valid → **202 `{"status": "accepted", "job_id": "..."}`** immediately, and
|
||||
the job runs in the background. 202-before-run means a long agent turn never
|
||||
trips the relay's HTTP timeout.
|
||||
- **At-most-once:** the agent claims the job with a store-level compare-and-set
|
||||
(`claim_job_for_fire`) before running. A relay/scheduler retry that arrives
|
||||
while the first fire is in flight (or after it completed) loses the claim and
|
||||
does not double-run.
|
||||
|
||||
---
|
||||
|
||||
## At-most-once & re-arm semantics
|
||||
|
||||
- **Recurring (cron/interval):** on fire, the agent advances `next_run_at`
|
||||
(under its store lock) as part of the claim, runs the job, then re-provisions
|
||||
a one-shot for the new `next_run_at`. A duplicate relay for the old `fire_at`
|
||||
finds the claim taken / time advanced and is dropped.
|
||||
- **One-shot (`30m`, `+90s`, etc.):** fires once; `mark_job_run` marks it
|
||||
completed. No re-arm.
|
||||
- **`repeat.times = N`:** `mark_job_run` deletes the job at the limit, so
|
||||
`get_job` returns `None` after the final fire → the agent does **not** re-arm
|
||||
→ the schedule stops cleanly with no orphaned one-shot.
|
||||
- **Multi-replica agents:** the store CAS makes the fire at-most-once across N
|
||||
gateway replicas sharing one `HERMES_HOME` — exactly one replica runs each
|
||||
fire.
|
||||
|
||||
## Reconcile (self-healing)
|
||||
|
||||
The agent reconciles desired (`jobs.json`) vs armed on:
|
||||
- `start()` (gateway boot / wake),
|
||||
- every successful job mutation (`on_jobs_changed`),
|
||||
- piggybacked after each fire (re-arm).
|
||||
|
||||
Reconcile arms missing/changed-time jobs and cancels orphans. A missed
|
||||
provision (transient NAS error) self-heals on the next reconcile. There is **no
|
||||
periodic wake** of a sleeping agent — that would negate scale-to-zero.
|
||||
|
||||
## Config (agent side)
|
||||
|
||||
All non-secret (`cron.chronos.*` in `config.yaml`); the agent holds no scheduler
|
||||
credentials. For hosted agents NAS sets these at provision time:
|
||||
|
||||
| key | meaning |
|
||||
|---|---|
|
||||
| `cron.provider` | `"chronos"` to activate (empty = built-in ticker) |
|
||||
| `cron.chronos.portal_url` | NAS base URL (also the expected JWT `iss`) |
|
||||
| `cron.chronos.callback_url` | the agent's own public base URL for NAS→agent fires |
|
||||
| `cron.chronos.expected_audience` | this agent's JWT `aud` (`agent:{instance_id}`) |
|
||||
| `cron.chronos.nas_jwks_url` | NAS JWKS for verifying the fire JWT |
|
||||
|
||||
If `callback_url` / `portal_url` is blank or the agent has no Nous login,
|
||||
`is_available()` returns False and the resolver falls back to the built-in
|
||||
in-process ticker — cron never loses its trigger.
|
||||
|
||||
## Escape hatch (not default)
|
||||
|
||||
The inbound `/api/cron/fire` verifier is pluggable (`get_fire_verifier()`). If
|
||||
relay volume through NAS ever saturates, a direct scheduler→agent mode with a
|
||||
per-job NAS-minted cron-key can replace the NAS-JWT verifier with **no change to
|
||||
the webhook handler**. NAS-mediated (this contract) is the default.
|
||||
@@ -0,0 +1,146 @@
|
||||
# Profile Builder — Dashboard-Native, Full-Featured Profile Creation
|
||||
|
||||
Status: design proposal (not yet implemented)
|
||||
Author: drafted for Teknium
|
||||
Supersedes: PR #31781 (prompt_toolkit `hermes profile wizard`)
|
||||
|
||||
## Why this, not the CLI wizard
|
||||
|
||||
PR #31781 added a keyboard-driven `hermes profile wizard` in the terminal.
|
||||
The decision is to **not** build the profile-creation experience in the CLI.
|
||||
The dashboard already owns mature, separate pages for every element a profile
|
||||
needs, and a profile is just a HERMES_HOME directory — so the dashboard is the
|
||||
right home for a full-featured builder, and it can reuse everything that
|
||||
already exists.
|
||||
|
||||
A profile = a full `~/.hermes/profiles/<name>/` directory with its own:
|
||||
- `config.yaml` — holds `model`/`provider`, `mcp_servers`, enabled skills
|
||||
- `skills/` — physical SKILL.md files (built-in seed + optional + hub installs)
|
||||
- `.env` — secrets
|
||||
- `SOUL.md` / `USER.md` — identity
|
||||
|
||||
So per-profile scoping of Model, MCPs, and Skills is **native** — no data-model
|
||||
change needed. The gap is purely UX: creation today is a thin modal
|
||||
(name + clone + model + description), and you can only compose skills/MCPs
|
||||
*after* the profile exists, by visiting other pages and remembering to scope
|
||||
them.
|
||||
|
||||
## What already exists (reuse, don't rebuild)
|
||||
|
||||
| Element | Existing page | Existing API | Profile-scopable? |
|
||||
|---|---|---|---|
|
||||
| Name / Description | ProfilesPage create modal | `POST /api/profiles` (`create_profile`) | yes (args) |
|
||||
| Model + Provider | ModelsPage | `_write_profile_model(profile_dir, …)` | yes — HERMES_HOME override, already wired into create endpoint |
|
||||
| MCPs | McpPage | `mcp_config._save_mcp_server` + `/api/mcp/catalog` | yes — wrap with HERMES_HOME override |
|
||||
| Skills (built-in/optional) | SkillsPage | `GET /api/skills`, `/api/skills/toggle` | yes — config write |
|
||||
| Skills (hub) | SkillsPage | `/api/skills/hub/search`, `/api/skills/hub/install` | **only via subprocess** — see seam #1 |
|
||||
|
||||
## Two architectural seams found while grounding this design
|
||||
|
||||
These are load-bearing — they change the implementation, not just the polish.
|
||||
|
||||
### Seam #1 — hub-skill install cannot use the HERMES_HOME override
|
||||
|
||||
`tools/skills_hub.py` binds `SKILLS_DIR = HERMES_HOME / "skills"` at **module
|
||||
import time**. The context-local `set_hermes_home_override()` swap (which makes
|
||||
`_write_profile_model` and the MCP write land in the target profile) does NOT
|
||||
retroactively rebind that already-imported module global. So a data-layer wrap
|
||||
of hub install would write into the dashboard's *own* active profile, not the
|
||||
new one.
|
||||
|
||||
The correct mechanism is the existing subprocess path: `_spawn_hermes_action`
|
||||
runs `python -m hermes_cli.main <subcommand>`, and `_apply_profile_override()`
|
||||
re-reads `sys.argv` at import in the fresh child. Prepend `-p <profile>`:
|
||||
|
||||
```python
|
||||
_spawn_hermes_action(["-p", profile, "skills", "install", identifier], "skills-install")
|
||||
```
|
||||
|
||||
A fresh subprocess re-imports `skills_hub` with the profile's HERMES_HOME bound
|
||||
from the start, so `SKILLS_DIR` resolves to `<profile>/skills/`. Correct by
|
||||
construction.
|
||||
|
||||
### Seam #2 — hub installs are async, so create cannot be fully atomic
|
||||
|
||||
Built-in/optional skill enabling and MCP writes are **synchronous config ops**
|
||||
and can be part of the create call. Hub installs are long-running git fetches
|
||||
spawned detached (`_spawn_hermes_action` returns a PID immediately). So the
|
||||
create flow is:
|
||||
|
||||
1. `create_profile()` — make the dir (synchronous)
|
||||
2. write model (synchronous, HERMES_HOME override)
|
||||
3. write selected MCP servers (synchronous, HERMES_HOME override)
|
||||
4. seed/enable selected built-in + optional skills (synchronous)
|
||||
5. spawn `hermes -p <profile> skills install <id>` per hub skill (async, returns PIDs)
|
||||
|
||||
Steps 1–4 commit before the response; step 5 returns a list of action PIDs the
|
||||
UI polls (same pattern as today's SkillsPage hub install). The builder's
|
||||
"Review → Create" returns `{ok, name, path, hub_installs: [{id, pid}]}` and the
|
||||
final screen shows live install progress for the hub skills.
|
||||
|
||||
## Proposed backend change (small, follows existing patterns)
|
||||
|
||||
Extend `ProfileCreate` and the create endpoint — no new endpoints, no rewrite:
|
||||
|
||||
```python
|
||||
class ProfileCreate(BaseModel):
|
||||
name: str
|
||||
clone_from: Optional[str] = None
|
||||
# Backward compatibility for older dashboard/desktop clients.
|
||||
clone_from_default: bool = False
|
||||
clone_all: bool = False
|
||||
no_skills: bool = False
|
||||
description: Optional[str] = None
|
||||
provider: Optional[str] = None
|
||||
model: Optional[str] = None
|
||||
# NEW — all optional, all best-effort post-create (profile already exists)
|
||||
mcp_servers: List[MCPServerCreate] = [] # synchronous, HERMES_HOME override
|
||||
builtin_skills: List[str] = [] # synchronous enable/seed
|
||||
hub_skills: List[str] = [] # async spawn, returns PIDs
|
||||
```
|
||||
|
||||
The endpoint already does best-effort post-create steps (`seed_profile_skills`,
|
||||
`_write_profile_model`). Add two more best-effort blocks (MCP write, hub-skill
|
||||
spawn) in the same style — a failure in any of them must not 500 the create,
|
||||
since the profile dir already exists and the user can fix it from the relevant
|
||||
page afterward. Mirror `_write_profile_model`'s HERMES_HOME-override helper for
|
||||
the MCP write (`_write_profile_mcp_servers(profile_dir, servers)`).
|
||||
|
||||
## Proposed frontend — dedicated builder page `/profiles/new`
|
||||
|
||||
A full page (not the cramped modal), stepped, each step reusing the existing
|
||||
page's component + API, targeted at the new profile:
|
||||
|
||||
```
|
||||
① Identity Name + Description (+ optional clone-from existing profile)
|
||||
② Model Provider + model picker (reuse ModelsPage picker)
|
||||
③ Skills Tabs: Built-in · Optional · Hub-search
|
||||
multi-select; "Start from default bundle" preset button
|
||||
④ MCPs Tabs: Catalog browse · Manual add (reuse McpPage form)
|
||||
⑤ Review Blueprint preview → Create
|
||||
→ progress screen for async hub installs
|
||||
```
|
||||
|
||||
Nothing writes to disk until ⑤.
|
||||
|
||||
## Open product decisions (need Teknium)
|
||||
|
||||
1. **Skills seeding default.** Fresh profiles auto-seed the default bundle
|
||||
today. In the builder, should the skill step **replace** the bundle (pick
|
||||
exactly what you want; offer a "start from default bundle" preset) or
|
||||
**augment** it? Recommendation: replace + preset button.
|
||||
|
||||
2. **Page vs richer modal.** Dedicated `/profiles/new` page (room to grow:
|
||||
SOUL editing, multi-agent fleets later) vs a bigger create modal on
|
||||
ProfilesPage. Recommendation: dedicated page — matches "full-featured / way
|
||||
more options."
|
||||
|
||||
## Verification plan (when built)
|
||||
|
||||
- Backend E2E with isolated HERMES_HOME: POST a full create body
|
||||
(name + model + 2 MCPs + 3 builtin skills + 1 hub skill), assert the new
|
||||
profile dir has the model in config.yaml, both MCP servers in config.yaml,
|
||||
the builtin skills enabled, and a spawned PID for the hub skill. Negative:
|
||||
a bad MCP entry must not 500 the create.
|
||||
- `cd web && npm run build` (no JS test suite in web/).
|
||||
- Targeted: `pytest tests/<web_server profile tests> -k profile_create`.
|
||||
Binary file not shown.
@@ -0,0 +1,39 @@
|
||||
# Multi-gateway deployment
|
||||
|
||||
Hermes supports multiple gateway processes running concurrently — one per profile
|
||||
(default, writer, admin, coder, researcher). Each gateway opens its own connection
|
||||
to platform APIs and delivers messages for its profile's subscribers.
|
||||
|
||||
## Single-dispatcher posture
|
||||
|
||||
Only one gateway owns the kanban dispatcher. The owning gateway keeps
|
||||
`kanban.dispatch_in_gateway: true` (the default); every other gateway sets it
|
||||
to `false`.
|
||||
|
||||
**Why this matters:** a gateway with `dispatch_in_gateway: true` opens per-board
|
||||
SQLite connections for both the dispatcher and the notifier watcher. Multiple
|
||||
gateways doing this concurrently multiplies the open file descriptors on each
|
||||
`kanban.db` and amplifies WAL `-shm` reader contention. Gating both paths on the
|
||||
same flag means exactly one process touches the kanban DBs.
|
||||
|
||||
## Configuration
|
||||
|
||||
On the dispatch-owning gateway (typically the `default` profile), no change is
|
||||
needed. On every other profile gateway, add to `~/.hermes/config.yaml`:
|
||||
|
||||
```yaml
|
||||
kanban:
|
||||
dispatch_in_gateway: false
|
||||
```
|
||||
|
||||
Or set the env var: `HERMES_KANBAN_DISPATCH_IN_GATEWAY=false`
|
||||
|
||||
## What each gateway does
|
||||
|
||||
| Gateway role | dispatch_in_gateway | Opens per-board DBs? | Runs dispatcher + notifier? |
|
||||
|---|---|---|---|
|
||||
| default (dispatch owner) | true (default) | yes | yes |
|
||||
| writer, admin, coder, etc. | false | no | no |
|
||||
|
||||
Non-dispatch gateways still deliver messages for their own platform adapters
|
||||
(Telegram, Discord, etc.) — they just don't poll kanban boards.
|
||||
@@ -0,0 +1,260 @@
|
||||
# Hermes Middleware
|
||||
|
||||
Hermes middleware is the behavior-changing companion to observer hooks.
|
||||
Observer hooks report what happened. Middleware can change what happens by
|
||||
rewriting a request before execution or by wrapping the execution callback
|
||||
itself.
|
||||
|
||||
This contract is intentionally backend-neutral. A plugin can use it for local
|
||||
policy, request shaping, tracing, adaptive routing, cache control, sandbox
|
||||
selection, or handoff to runtimes such as NeMo Relay without changing Hermes'
|
||||
planner, model provider adapters, tool registry, memory, or CLI UX.
|
||||
|
||||
With middleware enabled, plugins can:
|
||||
|
||||
- Rewrite LLM provider request kwargs before Hermes calls the provider.
|
||||
- Rewrite tool arguments before guardrails, approval checks, hooks, and tool
|
||||
execution see them.
|
||||
- Wrap the actual LLM execution callback while preserving Hermes retry,
|
||||
streaming, interrupt, and hook behavior.
|
||||
- Wrap the actual tool execution callback while preserving Hermes guardrails,
|
||||
approval, post-tool hooks, and tool-result transformation.
|
||||
|
||||
## Contract
|
||||
|
||||
Plugins register middleware from `register(ctx)`:
|
||||
|
||||
```python
|
||||
def register(ctx):
|
||||
ctx.register_middleware("llm_request", on_llm_request)
|
||||
ctx.register_middleware("llm_execution", on_llm_execution)
|
||||
ctx.register_middleware("tool_request", on_tool_request)
|
||||
ctx.register_middleware("tool_execution", on_tool_execution)
|
||||
```
|
||||
|
||||
Every middleware callback receives:
|
||||
|
||||
- `telemetry_schema_version`: currently `hermes.observer.v1`
|
||||
- `middleware_schema_version`: currently `hermes.middleware.v1`
|
||||
- Runtime context such as `session_id`, `task_id`, `turn_id`,
|
||||
`api_request_id`, `provider`, `model`, `api_mode`, `tool_name`, and
|
||||
`tool_call_id` when applicable.
|
||||
|
||||
Supported middleware kinds:
|
||||
|
||||
| Kind | Payload | Return shape | Purpose |
|
||||
| --- | --- | --- | --- |
|
||||
| `llm_request` | `request`, `original_request` | `{"request": {...}}` | Replace effective provider kwargs before provider execution. |
|
||||
| `tool_request` | `tool_name`, `args`, `original_args` | `{"args": {...}}` | Replace effective tool args before hooks, guardrails, approvals, and execution. |
|
||||
| `llm_execution` | `request`, `original_request`, `next_call` | Any provider response | Wrap or replace the actual provider call. |
|
||||
| `tool_execution` | `tool_name`, `args`, `original_args`, `next_call` | Any tool result | Wrap or replace the actual tool call. |
|
||||
|
||||
Request middleware can return optional trace fields:
|
||||
|
||||
```python
|
||||
return {
|
||||
"request": updated_request,
|
||||
"source": "my-plugin",
|
||||
"reason": "selected fallback model",
|
||||
}
|
||||
```
|
||||
|
||||
Hermes stores those trace entries in later observer hook payloads as
|
||||
`middleware_trace`.
|
||||
|
||||
Execution middleware receives a `next_call` callback. Call it to continue the
|
||||
chain:
|
||||
|
||||
```python
|
||||
def on_tool_execution(**kwargs):
|
||||
result = kwargs["next_call"](kwargs["args"])
|
||||
return result
|
||||
```
|
||||
|
||||
If multiple plugins register the same execution middleware kind, Hermes runs
|
||||
them as a nested chain in registration order. Middleware failures are fail-open:
|
||||
Hermes logs a warning and continues with the next middleware or the base
|
||||
runtime path.
|
||||
|
||||
## Execution Order
|
||||
|
||||
### LLM Calls
|
||||
|
||||
For each provider request, Hermes applies middleware in this order:
|
||||
|
||||
1. Build provider kwargs from the current conversation.
|
||||
2. Apply `llm_request` middleware.
|
||||
3. Emit `pre_api_request` observer hooks with the effective request.
|
||||
4. Run provider execution through `llm_execution` middleware.
|
||||
5. Emit `post_api_request` or `api_request_error` observer hooks.
|
||||
|
||||
Request middleware sees the full provider kwargs, including `messages` or
|
||||
Responses API `input`, model settings, tool definitions, stream options, and
|
||||
provider-specific options. Execution middleware receives the same effective
|
||||
request plus `next_call`.
|
||||
|
||||
### Tool Calls
|
||||
|
||||
For each tool call, Hermes applies middleware in this order:
|
||||
|
||||
1. Parse and coerce model-provided tool arguments.
|
||||
2. Apply `tool_request` middleware.
|
||||
3. Run the normal Hermes pre-execution path against the effective arguments:
|
||||
tool availability checks, observer block directives, guardrails, and
|
||||
approval checks.
|
||||
4. Run tool execution through `tool_execution` middleware.
|
||||
5. Emit `post_tool_call` observer hooks.
|
||||
6. Apply `transform_tool_result` hooks before the result is appended back into
|
||||
conversation context.
|
||||
|
||||
Tool request middleware runs before approval checks. Use it carefully: a
|
||||
rewritten path, command, or URL is the value downstream policy will evaluate.
|
||||
|
||||
## Enablement
|
||||
|
||||
Middleware only runs for enabled plugins. For a bundled plugin:
|
||||
|
||||
```bash
|
||||
hermes plugins enable <plugin-name>
|
||||
```
|
||||
|
||||
For isolated local testing, use one `HERMES_HOME` for plugin enablement and the
|
||||
agent run:
|
||||
|
||||
```bash
|
||||
export HERMES_HOME=/tmp/hermes-middleware-test
|
||||
mkdir -p "$HERMES_HOME"
|
||||
hermes plugins enable <plugin-name>
|
||||
hermes chat --query 'Reply exactly ok'
|
||||
```
|
||||
|
||||
For source checkouts, prefer the source command so the runtime sees plugins and
|
||||
middleware from the working tree:
|
||||
|
||||
```bash
|
||||
uv sync
|
||||
uv run hermes plugins enable <plugin-name>
|
||||
uv run hermes chat --query 'Reply exactly ok'
|
||||
```
|
||||
|
||||
## Generic Plugin Examples
|
||||
|
||||
The examples below are intentionally small. They show the middleware contract
|
||||
shape without depending on NeMo Relay.
|
||||
|
||||
### LLM Request Middleware
|
||||
|
||||
This plugin tags provider requests and records a middleware trace entry:
|
||||
|
||||
```python
|
||||
def register(ctx):
|
||||
ctx.register_middleware("llm_request", tag_llm_request)
|
||||
|
||||
|
||||
def tag_llm_request(**kwargs):
|
||||
request = dict(kwargs["request"])
|
||||
extra_body = dict(request.get("extra_body") or {})
|
||||
extra_body.setdefault("metadata", {})["hermes_middleware_demo"] = True
|
||||
request["extra_body"] = extra_body
|
||||
return {
|
||||
"request": request,
|
||||
"source": "middleware-demo",
|
||||
"reason": "tagged provider request",
|
||||
}
|
||||
```
|
||||
|
||||
The effective request is passed to `pre_api_request`, provider execution, and
|
||||
`post_api_request`.
|
||||
|
||||
### Tool Request Middleware
|
||||
|
||||
This plugin constrains `terminal` calls to a known working directory:
|
||||
|
||||
```python
|
||||
def register(ctx):
|
||||
ctx.register_middleware("tool_request", normalize_terminal_workdir)
|
||||
|
||||
|
||||
def normalize_terminal_workdir(**kwargs):
|
||||
if kwargs.get("tool_name") != "terminal":
|
||||
return None
|
||||
args = dict(kwargs["args"])
|
||||
args.setdefault("workdir", "/tmp/hermes-middleware-demo")
|
||||
return {
|
||||
"args": args,
|
||||
"source": "middleware-demo",
|
||||
"reason": "defaulted terminal workdir",
|
||||
}
|
||||
```
|
||||
|
||||
Because this runs before hooks and approvals, downstream telemetry and policy
|
||||
observe the rewritten `workdir`.
|
||||
|
||||
### LLM Execution Middleware
|
||||
|
||||
This plugin wraps the provider call and preserves the raw provider response:
|
||||
|
||||
```python
|
||||
import time
|
||||
|
||||
|
||||
def register(ctx):
|
||||
ctx.register_middleware("llm_execution", time_llm_execution)
|
||||
|
||||
|
||||
def time_llm_execution(**kwargs):
|
||||
started = time.monotonic()
|
||||
response = kwargs["next_call"](kwargs["request"])
|
||||
elapsed_ms = int((time.monotonic() - started) * 1000)
|
||||
print(f"llm_execution elapsed_ms={elapsed_ms}")
|
||||
return response
|
||||
```
|
||||
|
||||
Return the same response shape Hermes expects from the provider adapter. Do not
|
||||
wrap the response in a plugin-specific envelope unless the rest of the runtime
|
||||
expects that envelope.
|
||||
|
||||
### Tool Execution Middleware
|
||||
|
||||
This plugin wraps tool execution while preserving the tool result:
|
||||
|
||||
```python
|
||||
def register(ctx):
|
||||
ctx.register_middleware("tool_execution", annotate_tool_execution)
|
||||
|
||||
|
||||
def annotate_tool_execution(**kwargs):
|
||||
result = kwargs["next_call"](kwargs["args"])
|
||||
# Metrics, logging, or external routing can happen here.
|
||||
return result
|
||||
```
|
||||
|
||||
Execution middleware may call `next_call(modified_args)` to pass a changed
|
||||
payload to later middleware and the base tool dispatcher.
|
||||
|
||||
Plugin-specific examples should live with the plugin that owns the behavior.
|
||||
For NeMo Relay adaptive execution middleware, see
|
||||
[`plugins/observability/nemo_relay/README.md`](../../plugins/observability/nemo_relay/README.md).
|
||||
|
||||
## Safety Notes
|
||||
|
||||
- Middleware should be deterministic for the same input unless it is explicitly
|
||||
routing to a dynamic external system.
|
||||
- Request middleware should return complete replacement payloads, not partial
|
||||
patches.
|
||||
- Execution middleware should call `next_call(...)` exactly once unless it is
|
||||
intentionally short-circuiting execution.
|
||||
- If execution middleware raises before calling `next_call(...)`, Hermes treats
|
||||
that as middleware failure and continues with the remaining middleware chain
|
||||
and base execution.
|
||||
- If execution middleware calls `next_call(...)` successfully and then raises
|
||||
during post-processing, Hermes preserves the downstream result and does not
|
||||
run the provider or tool a second time.
|
||||
- If downstream provider or tool execution fails, middleware may let that error
|
||||
propagate or translate it deliberately. Hermes does not convert downstream
|
||||
failure into a successful `None` result.
|
||||
- Tool request middleware runs before approvals. If it mutates file paths,
|
||||
commands, URLs, or arguments, the mutated values are what guardrails and
|
||||
approvals evaluate.
|
||||
- Observer hooks remain the right place for read-only telemetry. Use middleware
|
||||
only when a plugin needs to alter or wrap behavior.
|
||||
@@ -0,0 +1,316 @@
|
||||
# Hermes Observer Hooks
|
||||
|
||||
Hermes observer hooks are the read-only telemetry contract for plugins that
|
||||
need to reconstruct agent execution without changing runtime behavior. This
|
||||
contract supports trace, metrics, audit, replay, and export integrations such
|
||||
as Langfuse, OpenTelemetry-style collectors, and NeMo Relay.
|
||||
|
||||
Observer hooks are intentionally backend-neutral. They expose stable lifecycle
|
||||
events, correlation IDs, sanitized payloads, timing, status, and error fields.
|
||||
They do not replace Hermes' planner, model providers, memory, tool registry,
|
||||
approval UX, CLI, gateway behavior, or execution semantics.
|
||||
|
||||
Behavior-changing request or execution wrappers are outside this observer
|
||||
contract. Observer hooks should report what happened; they should not replace
|
||||
provider requests, tool arguments, or execution callbacks.
|
||||
|
||||
## Contract
|
||||
|
||||
Plugins register observer callbacks from `register(ctx)`:
|
||||
|
||||
```python
|
||||
def register(ctx):
|
||||
ctx.register_hook("pre_api_request", on_pre_api_request)
|
||||
ctx.register_hook("post_api_request", on_post_api_request)
|
||||
ctx.register_hook("pre_tool_call", on_pre_tool_call)
|
||||
ctx.register_hook("post_tool_call", on_post_tool_call)
|
||||
```
|
||||
|
||||
Every hook callback receives keyword arguments. Plugins should accept
|
||||
`**kwargs` so additive fields remain backward-compatible:
|
||||
|
||||
```python
|
||||
def on_post_tool_call(**kwargs):
|
||||
tool_name = kwargs.get("tool_name")
|
||||
status = kwargs.get("status")
|
||||
result = kwargs.get("result")
|
||||
```
|
||||
|
||||
The plugin manager injects this field into every hook payload:
|
||||
|
||||
```text
|
||||
telemetry_schema_version = "hermes.observer.v1"
|
||||
```
|
||||
|
||||
Hook callbacks are fail-open. Hermes catches callback exceptions, logs a
|
||||
warning, and keeps the agent loop running.
|
||||
|
||||
Most observer hook return values are ignored. The exceptions are older
|
||||
behavior-affecting hooks:
|
||||
|
||||
| Hook | Return behavior |
|
||||
| --- | --- |
|
||||
| `pre_llm_call` | May return a string or `{"context": "..."}` to inject ephemeral context into the current user message. |
|
||||
| `pre_tool_call` | May return `{"action": "block", "message": "..."}` to block a tool before execution. |
|
||||
| `transform_tool_result` | May return a replacement tool result string after `post_tool_call`. |
|
||||
| `transform_llm_output` | May return a replacement final assistant text string. |
|
||||
|
||||
Telemetry plugins should treat these behavior-affecting returns as optional
|
||||
compatibility features, not as observability requirements.
|
||||
|
||||
## Correlation IDs
|
||||
|
||||
Observer payloads use stable IDs so plugins can join events without relying on
|
||||
callback order alone.
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `session_id` | Conversation/session identity. |
|
||||
| `task_id` | Task identity, especially useful for subagents and isolated execution. |
|
||||
| `turn_id` | User-turn identity shared by API attempts and tool calls in a turn. |
|
||||
| `api_request_id` | Opaque provider-attempt identity. Do not parse its string format. |
|
||||
| `api_call_count` | Numeric API attempt count within the agent loop. |
|
||||
| `tool_call_id` | Provider-supplied tool call ID when available. |
|
||||
| `parent_session_id` / `child_session_id` | Session link for delegated subagents. |
|
||||
| `parent_subagent_id` / `child_subagent_id` | Subagent link when available. |
|
||||
| `parent_turn_id` | Parent turn that spawned delegated work. |
|
||||
|
||||
Consumers should prefer explicit fields over parsing compound IDs. In
|
||||
particular, `api_request_id` is an opaque correlation value.
|
||||
|
||||
## Event Families
|
||||
|
||||
### Session Lifecycle
|
||||
|
||||
Session hooks describe conversation boundaries and resets:
|
||||
|
||||
| Hook | When it fires |
|
||||
| --- | --- |
|
||||
| `on_session_start` | A brand-new session starts after the system prompt is built. |
|
||||
| `on_session_end` | A `run_conversation` call ends, including interrupted or incomplete turns. |
|
||||
| `on_session_finalize` | CLI or gateway tears down an active session identity. |
|
||||
| `on_session_reset` | CLI or gateway moves from an old session identity to a new one. |
|
||||
|
||||
Common fields include `session_id`, `completed`, `interrupted`, `reason`,
|
||||
`old_session_id`, and `new_session_id` where available.
|
||||
|
||||
`on_session_end` is turn/run scoped. It is not necessarily the final lifetime
|
||||
boundary for a chat identity. Use `on_session_finalize` and `on_session_reset`
|
||||
for lifecycle cleanup that must happen once per session identity.
|
||||
|
||||
### Turn-Scoped LLM Hooks
|
||||
|
||||
These hooks frame the user turn, not individual provider API attempts:
|
||||
|
||||
| Hook | When it fires |
|
||||
| --- | --- |
|
||||
| `pre_llm_call` | Before the tool loop begins for a user turn. |
|
||||
| `post_llm_call` | After the turn completes with final assistant output. |
|
||||
|
||||
Common `pre_llm_call` fields include `session_id`, `turn_id`,
|
||||
`user_message`, `conversation_history`, `is_first_turn`, `model`, `platform`,
|
||||
and `sender_id`.
|
||||
|
||||
Common `post_llm_call` fields include `session_id`, `turn_id`,
|
||||
`user_message`, `assistant_response`, `conversation_history`, `model`, and
|
||||
`platform`.
|
||||
|
||||
Use request-scoped API hooks for LLM span telemetry. Use `pre_llm_call` and
|
||||
`post_llm_call` for turn-level context, compatibility, and final turn summary.
|
||||
|
||||
### Request-Scoped API Hooks
|
||||
|
||||
API hooks describe provider attempts inside the agent loop:
|
||||
|
||||
| Hook | When it fires |
|
||||
| --- | --- |
|
||||
| `pre_api_request` | Immediately before a provider API request. |
|
||||
| `post_api_request` | After a successful provider response. |
|
||||
| `api_request_error` | After a failed provider request or retryable error path. |
|
||||
|
||||
`pre_api_request` includes:
|
||||
|
||||
- identity: `session_id`, `task_id`, `turn_id`, `api_request_id`
|
||||
- runtime: `platform`, `model`, `provider`, `base_url`, `api_mode`
|
||||
- attempt metadata: `api_call_count`, `message_count`, `tool_count`,
|
||||
`approx_input_tokens`, `request_char_count`, `max_tokens`
|
||||
- timing: `started_at`
|
||||
- sanitized request payload: `request`
|
||||
|
||||
`post_api_request` includes the same identity/runtime fields plus:
|
||||
|
||||
- `api_duration`, `started_at`, `ended_at`
|
||||
- `finish_reason`, `message_count`, `response_model`
|
||||
- `usage`
|
||||
- `assistant_content_chars`, `assistant_tool_call_count`
|
||||
- sanitized response payload: `response`
|
||||
- compatibility object: `assistant_message`
|
||||
|
||||
`api_request_error` includes the same identity/runtime fields plus:
|
||||
|
||||
- `api_duration`, `started_at`, `ended_at`
|
||||
- `status_code`, `retry_count`, `max_retries`, `retryable`, `reason`
|
||||
- structured `error = {"type": ..., "message": ...}`
|
||||
- sanitized failed request payload: `request`
|
||||
|
||||
The sanitized `request`, `response`, and `error` fields are the canonical
|
||||
observer inputs for new consumers.
|
||||
|
||||
### Tool Lifecycle
|
||||
|
||||
Tool hooks describe individual tool calls:
|
||||
|
||||
| Hook | When it fires |
|
||||
| --- | --- |
|
||||
| `pre_tool_call` | Before guardrail-approved tool dispatch. |
|
||||
| `post_tool_call` | After tool dispatch, cancellation, block, or error completion. |
|
||||
| `transform_tool_result` | After `post_tool_call`, before the result is appended to model context. |
|
||||
|
||||
`pre_tool_call` includes `tool_name`, `args`, `task_id`, `session_id`,
|
||||
`tool_call_id`, `turn_id`, and `api_request_id`.
|
||||
|
||||
`post_tool_call` includes the same identity fields plus `result`,
|
||||
`duration_ms`, `status`, `error_type`, and `error_message`.
|
||||
|
||||
`status` is the observer-grade lifecycle outcome. Common values include:
|
||||
|
||||
| Status | Meaning |
|
||||
| --- | --- |
|
||||
| `ok` | Tool completed normally. |
|
||||
| `error` | Tool ran and returned or raised an error outcome. |
|
||||
| `blocked` | A `pre_tool_call` hook blocked execution. |
|
||||
| `cancelled` | Execution was cancelled before normal completion. |
|
||||
|
||||
`post_tool_call` is emitted for blocked and cancelled paths so telemetry
|
||||
plugins can close spans cleanly.
|
||||
|
||||
### Approval Lifecycle
|
||||
|
||||
Approval hooks describe dangerous-command approval prompts:
|
||||
|
||||
| Hook | When it fires |
|
||||
| --- | --- |
|
||||
| `pre_approval_request` | Before the approval request is shown or sent. |
|
||||
| `post_approval_response` | After the user responds or the request times out. |
|
||||
|
||||
Common fields include `command`, `description`, `pattern_key`,
|
||||
`pattern_keys`, `session_key`, and `surface`.
|
||||
|
||||
`post_approval_response` also includes `choice`, with values such as `once`,
|
||||
`session`, `always`, `deny`, and `timeout`.
|
||||
|
||||
Approval hooks are observer-only. Plugins cannot pre-answer or veto approvals
|
||||
from these hooks. To prevent a tool from reaching approval, use
|
||||
`pre_tool_call` blocking.
|
||||
|
||||
### Subagent Lifecycle
|
||||
|
||||
Subagent hooks describe delegated child-agent work:
|
||||
|
||||
| Hook | When it fires |
|
||||
| --- | --- |
|
||||
| `subagent_start` | A delegated child agent is created. |
|
||||
| `subagent_stop` | A delegated child agent returns or fails. |
|
||||
|
||||
`subagent_start` fields include `parent_session_id`, `parent_turn_id`,
|
||||
`parent_subagent_id`, `child_session_id`, `child_subagent_id`, `child_role`,
|
||||
and `child_goal`.
|
||||
|
||||
`subagent_stop` fields include parent/child session IDs, role/status fields,
|
||||
`child_summary`, and `duration_ms`.
|
||||
|
||||
Observers can use these hooks to model nested trajectories while keeping child
|
||||
agent execution linked to the parent turn that spawned it.
|
||||
|
||||
## Payload Safety
|
||||
|
||||
Observer payloads are designed for telemetry consumers, not raw object access.
|
||||
New consumers should use the sanitized API payloads:
|
||||
|
||||
- `pre_api_request.request`
|
||||
- `post_api_request.response`
|
||||
- `api_request_error.request`
|
||||
- `api_request_error.error`
|
||||
|
||||
Sanitization converts provider objects to JSON-compatible structures, bounds
|
||||
large payloads, redacts sensitive keys, and avoids exposing raw response
|
||||
objects in sanitized fields.
|
||||
|
||||
Legacy compatibility fields such as `request_messages`, `conversation_history`,
|
||||
and `assistant_message` may still be present for existing plugins. New
|
||||
observability consumers should prefer the sanitized payloads.
|
||||
|
||||
## Performance
|
||||
|
||||
The default uninstrumented path should stay cheap. Expensive request/response
|
||||
payload construction is gated behind `has_hook(...)`, so Hermes only builds
|
||||
sanitized API telemetry payloads when at least one plugin registered the
|
||||
relevant hook.
|
||||
|
||||
Plugin authors should preserve this property:
|
||||
|
||||
- Register only hooks the plugin actually consumes.
|
||||
- Avoid deep-copying or re-sanitizing already sanitized payloads.
|
||||
- Keep hook callbacks fast and fail-open.
|
||||
- Offload network export or batch writes when practical.
|
||||
|
||||
## Writing An Observer Plugin
|
||||
|
||||
Minimal observer plugin:
|
||||
|
||||
```python
|
||||
def register(ctx):
|
||||
ctx.register_hook("pre_api_request", on_pre_api_request)
|
||||
ctx.register_hook("post_api_request", on_post_api_request)
|
||||
ctx.register_hook("pre_tool_call", on_pre_tool_call)
|
||||
ctx.register_hook("post_tool_call", on_post_tool_call)
|
||||
|
||||
|
||||
def on_pre_api_request(**kwargs):
|
||||
start_llm_span(
|
||||
request_id=kwargs.get("api_request_id"),
|
||||
turn_id=kwargs.get("turn_id"),
|
||||
request=kwargs.get("request"),
|
||||
model=kwargs.get("model"),
|
||||
)
|
||||
|
||||
|
||||
def on_post_api_request(**kwargs):
|
||||
finish_llm_span(
|
||||
request_id=kwargs.get("api_request_id"),
|
||||
response=kwargs.get("response"),
|
||||
usage=kwargs.get("usage"),
|
||||
duration=kwargs.get("api_duration"),
|
||||
)
|
||||
|
||||
|
||||
def on_pre_tool_call(**kwargs):
|
||||
start_tool_span(
|
||||
call_id=kwargs.get("tool_call_id"),
|
||||
name=kwargs.get("tool_name"),
|
||||
args=kwargs.get("args"),
|
||||
)
|
||||
|
||||
|
||||
def on_post_tool_call(**kwargs):
|
||||
finish_tool_span(
|
||||
call_id=kwargs.get("tool_call_id"),
|
||||
result=kwargs.get("result"),
|
||||
status=kwargs.get("status"),
|
||||
duration_ms=kwargs.get("duration_ms"),
|
||||
)
|
||||
```
|
||||
|
||||
Use `session_id`, `turn_id`, `api_request_id`, and `tool_call_id` for span
|
||||
correlation. Use subagent and approval hooks when the export format supports
|
||||
nested agent work or security lifecycle events.
|
||||
|
||||
## Existing Consumers
|
||||
|
||||
The bundled Langfuse plugin demonstrates direct hook-based observability for
|
||||
turns, provider requests, and tool calls.
|
||||
|
||||
The bundled NeMo Relay plugin maps the same generic observer contract to NeMo
|
||||
Relay scopes, LLM spans, tool spans, marks, ATOF streams, and ATIF exports.
|
||||
NeMo Relay-specific configuration and examples live in
|
||||
[`plugins/observability/nemo_relay/README.md`](../../plugins/observability/nemo_relay/README.md).
|
||||
@@ -0,0 +1,240 @@
|
||||
---
|
||||
title: "fix: Prevent Telegram streamed replies from ending after first overflow chunk"
|
||||
status: active
|
||||
date: 2026-06-09
|
||||
type: fix
|
||||
target_repo: hermes-agent
|
||||
origin: user-reported Telegram topic screenshot
|
||||
---
|
||||
|
||||
# fix: Prevent Telegram streamed replies from ending after first overflow chunk
|
||||
|
||||
## Summary
|
||||
|
||||
Fix a Telegram gateway bug where a long streamed assistant reply can appear to stop mid-answer in a topic after the first overflow chunk. The reported screenshot shows a long Hermes response in the `Nehemiah - Coding` Telegram topic ending at `- The visible tool-call summary`, followed by the user noting that the previous message did not finish streaming to that Telegram topic.
|
||||
|
||||
The plan targets the streamed edit overflow path, not general model generation. A completed assistant response must either reach Telegram in full across all continuation messages or leave enough state for the gateway fallback path to deliver the remaining content instead of marking the turn complete after a partial delivery.
|
||||
|
||||
---
|
||||
|
||||
## Problem Frame
|
||||
|
||||
Telegram limits message text to 4096 UTF-16 code units. Hermes streams gateway responses by editing a message and, when a streamed message grows past the limit, splitting the overflow into additional Telegram messages. The adapter already has a split-and-deliver path for oversized edits, but the partial-continuation failure contract is weak: if chunk 1 is edited successfully and a later continuation fails, the adapter can still report success for the operation. The stream consumer may then mark the final response delivered even though the visible topic only contains the first part.
|
||||
|
||||
This is especially visible in Telegram forum topics because a long final response can be split below tool-progress bubbles, and a missing continuation looks exactly like the stream stopped mid-answer.
|
||||
|
||||
---
|
||||
|
||||
## Requirements
|
||||
|
||||
- R1. Long streamed Telegram replies must preserve all final content across overflow chunks.
|
||||
- R2. If any continuation chunk fails after the first overflow edit lands, the gateway must not mark the final response as fully delivered.
|
||||
- R3. Continuation chunks must remain routed to the same Telegram topic/thread as the original response.
|
||||
- R4. The fix must avoid duplicate full-answer sends when all overflow chunks were delivered successfully.
|
||||
- R5. Tests must cover the reported failure shape: a final streamed reply that exceeds Telegram's limit, succeeds on the first edit, fails on a continuation, and must not be treated as complete.
|
||||
|
||||
---
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
- Treat overflow delivery as all-or-not-complete. `_edit_overflow_split` should only return a successful final-delivery result when every planned chunk reaches Telegram. Partial delivery is a distinct outcome that downstream code can recover from.
|
||||
- Carry partial-overflow metadata through `SendResult.raw_response` rather than adding a new public dataclass field unless implementation proves the existing result shape is insufficient. The stream consumer already inspects `SendResult` after adapter edits, so a small raw response contract can keep the change contained.
|
||||
- Make the stream consumer responsible for final-delivery truth. The adapter knows which chunks landed, but the consumer owns `_final_response_sent`, `_final_content_delivered`, `_fallback_prefix`, and fallback final-send behaviour.
|
||||
- Keep routing inside Telegram adapter helpers. Continuation sends should continue to use `_thread_kwargs_for_send(...)` with metadata-derived `message_thread_id` and reply anchors so forum topic behaviour stays consistent.
|
||||
|
||||
---
|
||||
|
||||
## High-Level Technical Design
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as GatewayStreamConsumer
|
||||
participant T as TelegramAdapter.edit_message
|
||||
participant B as Telegram Bot API
|
||||
|
||||
C->>T: finalize/edit long accumulated response
|
||||
T->>B: edit original message with chunk 1
|
||||
loop remaining chunks
|
||||
T->>B: send continuation in same topic/thread
|
||||
end
|
||||
alt all chunks delivered
|
||||
T-->>C: success, last message id, continuation ids
|
||||
C->>C: mark final response delivered
|
||||
else any continuation failed
|
||||
T-->>C: partial overflow failure with delivered prefix metadata
|
||||
C->>C: do not mark final delivered
|
||||
C->>B: fallback sends missing tail or full final response safely
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Units
|
||||
|
||||
### U1. Add a partial-overflow contract for Telegram edit splits
|
||||
|
||||
**Goal:** Make `TelegramAdapter._edit_overflow_split` distinguish complete overflow delivery from partial delivery.
|
||||
|
||||
**Requirements:** R1, R2, R4
|
||||
|
||||
**Dependencies:** None
|
||||
|
||||
**Files:**
|
||||
- `gateway/platforms/telegram.py`
|
||||
- `tests/gateway/test_telegram_send.py` or the existing Telegram adapter test module that already covers `edit_message` overflow behaviour
|
||||
|
||||
**Approach:**
|
||||
- Keep the successful path unchanged when every chunk is delivered: return `SendResult(success=True, message_id=<last chunk>, continuation_message_ids=(...))`.
|
||||
- When a continuation fails after the first edit, return a result that clearly indicates partial delivery instead of plain success. Prefer `success=False`, `retryable=True`, and `raw_response` metadata such as delivered chunk count, total chunk count, last delivered message id, and the visible delivered prefix.
|
||||
- Preserve logging, but do not rely on logs as the only signal. The caller must be able to tell partial delivery happened.
|
||||
- Ensure the first edited chunk and all successful continuation chunks still include the existing Markdown/plain-text fallback behaviour.
|
||||
|
||||
**Patterns to follow:**
|
||||
- Existing overflow handling in `TelegramAdapter.edit_message` and `_edit_overflow_split`.
|
||||
- Existing `SendResult` semantics in `gateway/platforms/base.py`, especially `retryable`, `raw_response`, and `continuation_message_ids`.
|
||||
|
||||
**Test scenarios:**
|
||||
- Oversized finalized edit where all continuations succeed returns success, the last continuation id, and all continuation ids.
|
||||
- Oversized finalized edit where the first continuation send fails returns a partial-overflow failure and does not report success.
|
||||
- Oversized finalized edit where one continuation succeeds and a later continuation fails reports the last delivered continuation id and delivered count in raw metadata.
|
||||
- A continuation MarkdownV2 formatting failure still retries plain text before being treated as a delivery failure.
|
||||
|
||||
**Verification:** Adapter tests prove complete overflow remains successful and partial overflow is observable by the caller.
|
||||
|
||||
### U2. Teach the stream consumer to recover from partial overflow
|
||||
|
||||
**Goal:** Ensure a partial Telegram overflow does not set `_final_response_sent` or `_final_content_delivered` unless the full response reached the user.
|
||||
|
||||
**Requirements:** R1, R2, R4, R5
|
||||
|
||||
**Dependencies:** U1
|
||||
|
||||
**Files:**
|
||||
- `gateway/stream_consumer.py`
|
||||
- `tests/gateway/test_stream_consumer.py` or a focused new `tests/gateway/test_stream_consumer_telegram_overflow.py`
|
||||
|
||||
**Approach:**
|
||||
- In `_send_or_edit`, when `adapter.edit_message(...)` returns a partial-overflow failure, update consumer state to reflect the last visible prefix/message and enter fallback delivery for the missing content.
|
||||
- Avoid treating `_already_sent` as final delivery. A partial visible message can be true while final delivery is false.
|
||||
- Use the delivered-prefix metadata if available so `_send_fallback_final(...)` sends only the missing tail. If implementation finds the prefix is unreliable after Markdown formatting, prefer sending the complete final response as a fresh fallback message rather than silently dropping the tail.
|
||||
- Keep the existing success handling for `continuation_message_ids` when the adapter delivered all chunks.
|
||||
|
||||
**Patterns to follow:**
|
||||
- Existing fallback mode in `GatewayStreamConsumer._send_or_edit` and `_send_fallback_final`.
|
||||
- Existing comments around `_final_response_sent`, `_final_content_delivered`, and `_fallback_prefix` for prior partial-delivery regressions.
|
||||
|
||||
**Test scenarios:**
|
||||
- A final streamed response that overflows and receives a complete-success edit split sets final-delivery flags and does not invoke fallback.
|
||||
- A final streamed response whose adapter reports partial overflow does not set final-delivery flags immediately.
|
||||
- After partial overflow, fallback delivery sends the remaining tail and then marks final content delivered only if the fallback send succeeds.
|
||||
- If fallback delivery also fails, the consumer leaves final-delivery false so the gateway's non-streaming final-send safety path can still run.
|
||||
|
||||
**Verification:** Stream consumer tests reproduce the screenshot shape by simulating first chunk visible and continuation failure, then assert the final answer is not suppressed.
|
||||
|
||||
### U3. Preserve Telegram topic/thread routing for overflow and fallback continuations
|
||||
|
||||
**Goal:** Ensure overflow recovery messages land in the same Telegram forum topic or DM topic fallback context.
|
||||
|
||||
**Requirements:** R3
|
||||
|
||||
**Dependencies:** U1, U2
|
||||
|
||||
**Files:**
|
||||
- `gateway/platforms/telegram.py`
|
||||
- `gateway/stream_consumer.py`
|
||||
- `tests/gateway/test_stream_consumer_thread_routing.py`
|
||||
- Relevant Telegram adapter routing tests, if existing coverage is closer there
|
||||
|
||||
**Approach:**
|
||||
- Keep passing `metadata` through every overflow continuation and fallback send.
|
||||
- Keep reply anchors where valid, but do not let a missing reply anchor drop the `message_thread_id` for normal forum topics.
|
||||
- For private DM topic fallback metadata, preserve the existing stricter anchor behaviour documented in the adapter comments.
|
||||
|
||||
**Patterns to follow:**
|
||||
- `TelegramAdapter._thread_kwargs_for_send(...)`.
|
||||
- Existing tests around Telegram topic recovery and stream consumer thread routing.
|
||||
|
||||
**Test scenarios:**
|
||||
- Overflow continuations include `message_thread_id` for a forum topic.
|
||||
- A continuation retry after `reply message not found` keeps forum topic routing when allowed.
|
||||
- Partial-overflow fallback sends receive the same metadata passed to the original stream consumer.
|
||||
|
||||
**Verification:** Thread-routing assertions inspect fake bot calls and confirm all continuation/fallback messages carry the expected topic metadata.
|
||||
|
||||
### U4. Add issue evidence and PR body traceability
|
||||
|
||||
**Goal:** Make the upstream issue and PR clearly trace the user-visible bug and verification evidence.
|
||||
|
||||
**Requirements:** R5
|
||||
|
||||
**Dependencies:** U1, U2, U3
|
||||
|
||||
**Files:**
|
||||
- GitHub issue body created via `gh issue create`
|
||||
- PR body using `.github/PULL_REQUEST_TEMPLATE.md`
|
||||
|
||||
**Approach:**
|
||||
- Create a GitHub issue with the screenshot evidence: the long message in the `Nehemiah - Coding` Telegram topic stops at `- The visible tool-call summary`, and the user's reply says the previous message did not finish streaming to that Telegram topic.
|
||||
- Reference affected component as Gateway and platform as Telegram.
|
||||
- In the PR body, link the issue with `Fixes #...`, describe the split-delivery contract change, and include the screenshot or attach it if GitHub upload is available.
|
||||
- Follow `CONTRIBUTING.md` and the repository PR template exactly.
|
||||
|
||||
**Patterns to follow:**
|
||||
- `.github/ISSUE_TEMPLATE/bug_report.yml`
|
||||
- `.github/PULL_REQUEST_TEMPLATE.md`
|
||||
|
||||
**Test scenarios:**
|
||||
- Test expectation: none, this is tracker and PR documentation work.
|
||||
|
||||
**Verification:** The GitHub issue exists with screenshot evidence or an explicit screenshot reference, and the PR body links the issue and lists the tests run.
|
||||
|
||||
---
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
### In Scope
|
||||
|
||||
- Telegram streamed response overflow splitting and recovery.
|
||||
- Stream consumer final-delivery truth for partial overflow delivery.
|
||||
- Topic/thread metadata preservation for overflow and fallback continuation sends.
|
||||
- Focused unit tests around adapter and stream consumer behaviour.
|
||||
|
||||
### Out of Scope
|
||||
|
||||
- Changing model streaming semantics in `run_agent.py`.
|
||||
- Reworking Telegram draft streaming, which is DM-only and not the forum-topic path in the screenshot.
|
||||
- Changing general platform message splitting for Discord, Slack, WhatsApp, or Matrix unless a shared helper must be corrected for the Telegram fix.
|
||||
- Altering tool-progress display settings or terminal progress rendering.
|
||||
|
||||
### Deferred to Follow-Up Work
|
||||
|
||||
- Broader observability for gateway delivery completeness across all messaging platforms.
|
||||
- A user-facing resend/recover command for a previous truncated response.
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- Risk: fallback recovery duplicates already-visible first chunks. Mitigation: use delivered-prefix metadata where reliable and add tests for no-duplicate complete-success behaviour.
|
||||
- Risk: preserving forum topic routing while dropping invalid reply anchors is easy to regress. Mitigation: include fake bot call assertions for `message_thread_id` and reply behaviour.
|
||||
- Risk: MarkdownV2 formatting can alter visible/raw prefix comparisons. Mitigation: keep fallback conservative; duplicate content is preferable to silently missing content, but tests should keep the common path tail-only.
|
||||
|
||||
---
|
||||
|
||||
## Sources & Research
|
||||
|
||||
- User-provided screenshot at `/root/.hermes/image_cache/img_f664e68f6ddf.jpg`.
|
||||
- `gateway/stream_consumer.py` streamed edit, overflow, fallback, and final-delivery state handling.
|
||||
- `gateway/platforms/telegram.py` Telegram send/edit overflow splitting and topic routing helpers.
|
||||
- `gateway/platforms/base.py` `SendResult` contract and shared message chunking helper.
|
||||
- `tests/gateway/test_stream_consumer.py`, `tests/gateway/test_stream_consumer_thread_routing.py`, and Telegram adapter tests for focused regression coverage.
|
||||
|
||||
---
|
||||
|
||||
## Verification Strategy
|
||||
|
||||
- Run focused Telegram adapter overflow tests.
|
||||
- Run focused stream consumer overflow/fallback tests.
|
||||
- Run topic-routing tests affected by metadata changes.
|
||||
- Run the gateway test subset around Telegram send/edit, stream consumer, and run progress if touched.
|
||||
- Before PR creation, ensure `git diff` contains only the plan, implementation, tests, and PR/issue-relevant documentation for this bug.
|
||||
@@ -0,0 +1,54 @@
|
||||
# RCA: SSL CA cert bundle corruption after `hermes update`
|
||||
|
||||
**Status:** resolved by `fix(ssl): surface broken CA bundles before provider calls`
|
||||
**Severity:** P2 — degrades the agent into opaque provider/client failures until the user repairs deps or CA configuration.
|
||||
|
||||
## Summary
|
||||
|
||||
A partial `hermes update`, interrupted venv repair, or stale CA-bundle environment variable can leave Python TLS configuration pointing at a missing, empty, or unloadable CA bundle. The first outbound HTTPS client creation or request can then fail with a raw `FileNotFoundError: [Errno 2] No such file or directory` or a low-level SSL error that does not name the broken CA path.
|
||||
|
||||
## Root cause
|
||||
|
||||
Hermes uses OpenAI/httpx and requests-based clients for provider calls, model metadata, gateway delivery, and web tools. Those clients inherit CA bundle settings from:
|
||||
|
||||
- `HERMES_CA_BUNDLE`
|
||||
- `SSL_CERT_FILE`
|
||||
- `REQUESTS_CA_BUNDLE`
|
||||
- `CURL_CA_BUNDLE`
|
||||
- the bundled `certifi` package's `cacert.pem`
|
||||
|
||||
When the venv is partially refreshed, or when one of those env vars points at a file that no longer exists, provider client construction can fail before Hermes has enough context to produce a useful message.
|
||||
|
||||
## Fix
|
||||
|
||||
`agent/ssl_guard.py` validates CA bundle configuration before the OpenAI-compatible provider client is created in `agent/agent_init.py`. It:
|
||||
|
||||
1. Checks explicit CA bundle env vars and reports the exact broken variable/path,
|
||||
2. Verifies `certifi` is importable,
|
||||
3. Verifies `certifi.where()` points at an existing file of plausible size,
|
||||
4. Builds an `ssl.SSLContext` from each checked bundle,
|
||||
5. Raises a typed `SSLConfigurationError` with a repair hint before httpx/OpenAI can raise a raw low-level error.
|
||||
|
||||
`hermes_cli doctor` exposes the same check under `SSL / CA Certificates`, so users can diagnose the problem without starting a model session.
|
||||
|
||||
## Recovery
|
||||
|
||||
When the guard fires during agent init, the user sees a message like:
|
||||
|
||||
```text
|
||||
Failed to initialize OpenAI client: SSL_CERT_FILE points to a missing CA bundle: C:\path\to\missing\cacert.pem
|
||||
Repair: python -m pip install --force-reinstall certifi openai httpx
|
||||
If you configured a custom corporate CA bundle, fix or unset the broken CA bundle environment variable.
|
||||
```
|
||||
|
||||
For a normal corrupted Hermes venv, reinstall the affected client dependencies:
|
||||
|
||||
```bash
|
||||
python -m pip install --force-reinstall certifi openai httpx
|
||||
```
|
||||
|
||||
For a custom/corporate CA setup, fix the env var so it points at a real PEM bundle, or unset it if Hermes should use the bundled `certifi` store.
|
||||
|
||||
## Environment escape hatch
|
||||
|
||||
Set `HERMES_SKIP_SSL_GUARD=1` to bypass the preflight check. This is intended only for sandboxed or managed-trust environments where the Python CA path looks unusual but downstream clients are known to work.
|
||||
@@ -0,0 +1,568 @@
|
||||
# Relay ↔ Connector Contract (v1, EXPERIMENTAL)
|
||||
|
||||
> **Status:** EXPERIMENTAL. This contract MAY CHANGE without a deprecation
|
||||
> cycle until at least two real Class-1 platforms (Discord + Telegram) have
|
||||
> validated it. Evolution during the experimental phase is **additive-only**,
|
||||
> gated by `contract_version`. A breaking change updates both repos in lockstep.
|
||||
|
||||
This document is the formal interface between the **Hermes gateway** (Python,
|
||||
`gateway/relay/`) and the **connector** (Node/TypeScript,
|
||||
`NousResearch/gateway-gateway`). The connector implementer's first action is to
|
||||
read this file.
|
||||
|
||||
The gateway runs a generic `RelayAdapter` that dials **out** to the connector,
|
||||
receives a `CapabilityDescriptor` at handshake, then exchanges normalized
|
||||
`MessageEvent`s (inbound) and actions (outbound) over a per-turn bidirectional
|
||||
WebSocket. The gateway never learns which concrete platform is fronting it; the
|
||||
connector owns all platform-specific socket/identity logic.
|
||||
|
||||
---
|
||||
|
||||
## 1. Handshake
|
||||
|
||||
1. Gateway opens the transport (`connect`).
|
||||
2. Gateway calls `handshake()`; connector returns a `CapabilityDescriptor`
|
||||
(section 2) describing the platform this adapter instance fronts.
|
||||
3. Gateway configures the adapter from the descriptor (char limit, length unit,
|
||||
draft/edit/thread/markdown capabilities) and registers an inbound handler.
|
||||
4. Connector then streams inbound events and accepts outbound actions.
|
||||
|
||||
`contract_version` (currently `1`) is carried in the descriptor. The gateway
|
||||
ignores unknown descriptor fields (forward-compat) and fills missing optional
|
||||
fields from defaults.
|
||||
|
||||
---
|
||||
|
||||
## 2. CapabilityDescriptor (handshake payload)
|
||||
|
||||
JSON object. Source of truth: `gateway/relay/descriptor.py`.
|
||||
|
||||
| Field | Type | Required | Meaning |
|
||||
| --- | --- | --- | --- |
|
||||
| `contract_version` | int | yes | Contract version (additive-only within a version). |
|
||||
| `platform` | string | yes | Platform name (e.g. `"discord"`, `"telegram"`). |
|
||||
| `label` | string | yes | Human-readable label. |
|
||||
| `max_message_length` | int | yes | Char limit; gateway exposes as `MAX_MESSAGE_LENGTH`. 0 → treat as 4096. |
|
||||
| `supports_draft_streaming` | bool | yes | Native draft-streaming preview support. |
|
||||
| `supports_edit` | bool | yes | Edit-based streaming possible; if false, consumer degrades to one-message-per-segment. |
|
||||
| `supports_threads` | bool | yes | `create_handoff_thread` capability. |
|
||||
| `markdown_dialect` | string | yes | `"plain"`, `"markdown_v2"`, `"discord"`, … (drives `supports_code_blocks`). |
|
||||
| `len_unit` | string | yes | `"chars"` (builtin len) or `"utf16"` (Telegram UTF-16 code units). |
|
||||
| `emoji` | string | no | Display emoji (default 🔌). |
|
||||
| `platform_hint` | string | no | System-prompt platform hint. |
|
||||
| `pii_safe` | bool | no | Redact PII in session descriptions. |
|
||||
|
||||
Most fields are a projection of the gateway's existing `PlatformEntry`; the
|
||||
runtime-only fields (`len_unit`, `supports_*`, `markdown_dialect`) come from the
|
||||
live platform adapter's capability methods.
|
||||
|
||||
---
|
||||
|
||||
## 3. Inbound: `MessageEvent` envelope
|
||||
|
||||
The connector normalizes each platform wire event into a `MessageEvent`
|
||||
(`gateway/platforms/base.py`) and delivers it to the gateway. **Inbound is
|
||||
delivered over the gateway's OUTBOUND `/relay` WebSocket** (see the transport
|
||||
note below) — the connector pushes an `inbound` frame down the socket the
|
||||
gateway already dialed. The gateway keys the session via `build_session_key()`
|
||||
from the embedded `SessionSource` — so populating the right discriminators is
|
||||
the single highest-correctness responsibility of the connector.
|
||||
|
||||
### Inbound transport (WS back-channel, not HTTP)
|
||||
|
||||
The gateway dials **out** to the connector's `/relay` WebSocket for the
|
||||
handshake + outbound actions (§4) + its own `/stop` egress (§5). Inbound rides
|
||||
the **same socket** in the other direction: the connector pushes an `inbound`
|
||||
frame (and `interrupt_inbound` for §5) down the gateway's outbound WS. There is
|
||||
**no gateway-side inbound HTTP endpoint** — a gateway need not (and, when hosted,
|
||||
cannot) expose any inbound port; everything flows over the connection it
|
||||
initiated.
|
||||
|
||||
**Multi-instance routing.** The connector instance that owns a platform's socket
|
||||
(and thus produces inbound events) is generally **not** the instance the gateway
|
||||
dialed its outbound WS into. The producing instance therefore publishes the
|
||||
event on the connector's internal **relay bus** (Redis pub/sub; `RelayBus` in
|
||||
`src/core/relayBus.ts`) keyed by tenant. Every connector instance subscribes and
|
||||
routes each message to its **local** sessions for that tenant
|
||||
(`RelayServer.routeBusMessage`); the single instance that actually holds the
|
||||
gateway's socket delivers it, and instances with no local session for the tenant
|
||||
no-op. Cross-instance delivery is thus an in-cluster Redis hop, not a public
|
||||
HTTP call.
|
||||
|
||||
Frames (connector → gateway, over the WS):
|
||||
|
||||
- `{"type":"inbound", "event": <MessageEvent>, "bufferId"?}`
|
||||
- `{"type":"interrupt_inbound", "session_key", "chat_id"}` (§5)
|
||||
- `{"type":"passthrough_forward", "forward": <PassthroughForward>, "bufferId"?}` (§5.1)
|
||||
|
||||
`PassthroughForward` is the wire form of a forwarded passthrough-plane request
|
||||
(Class-2/3 webhooks — Discord interactions, Twilio): `{platform, botId, method,
|
||||
path, headers: [[k,v],…], bodyB64}`. The body is base64-encoded so arbitrary
|
||||
bytes survive the newline-delimited-JSON transport; the gateway base64-decodes
|
||||
back to the exact bytes the connector forwarded (the connector already verified
|
||||
the provider signature and stripped any shared-identity credential at the edge —
|
||||
§6 — so the gateway re-processes a sanitized, token-free body and acts on it via
|
||||
the token-less `follow_up` path). See §3.1.
|
||||
|
||||
**Trust.** The WS upgrade is authenticated with the gateway's per-gateway secret
|
||||
(§6.1), so the channel is trusted end to end — inbound frames are not separately
|
||||
HMAC-signed (the authenticated socket subsumes the per-delivery origin proof the
|
||||
old HTTP path needed). The relay-bus hop is inside the connector trust domain
|
||||
(same as the lease/buffer/capability stores).
|
||||
|
||||
> Earlier drafts of this contract delivered inbound over a signed **HTTP POST**
|
||||
> to a `gatewayEndpoint` (`HttpGatewayDelivery` + a gateway-side
|
||||
> `inbound_receiver`), HMAC-signed with a per-tenant delivery key. That required
|
||||
> every gateway to expose a reachable inbound URL — impossible for hosted
|
||||
> gateways, which have no public IP. The WS back-channel above replaces it; the
|
||||
> per-tenant delivery key is retained at provision for forward-compat but is no
|
||||
> longer used for inbound. The **passthrough plane** (Class-2/3 webhooks like
|
||||
> Discord interactions / Twilio) historically still used `gatewayEndpoint` for
|
||||
> its post-ACK forward; Phase 5 §5.1 moves that forward onto the WS too (the
|
||||
> `passthrough_forward` frame above), so a hosted gateway needs zero public
|
||||
> inbound surface and `gatewayEndpoint` is retired once the cutover lands.
|
||||
|
||||
### 3.1 Passthrough-plane forward (§5.1)
|
||||
|
||||
The passthrough plane answers the provider's latency-critical ACK at the
|
||||
connector EDGE (e.g. Discord's deferred interaction response within ~3s), then
|
||||
does a **fire-and-forget** forward of the real request to the gateway. That
|
||||
forward needs no response back (the provider was already satisfied), so it rides
|
||||
the same outbound WS as `inbound` via a `passthrough_forward` frame rather than
|
||||
an HTTP POST. The gateway processes the decoded request through its normal agent
|
||||
path (a Discord interaction is decoded to a `MessageEvent` and handled like a
|
||||
message; the reply egresses over the outbound / `follow_up` path). `bufferId` is
|
||||
present when the forward was buffered (Phase 5 §5.3 buffered-only flip) and the
|
||||
gateway acks it after durable handoff.
|
||||
|
||||
|
||||
|
||||
### SessionSource fields (the wire surface)
|
||||
|
||||
Source of truth: `SessionSource.to_dict()` in `gateway/session.py`. These are
|
||||
every key the gateway accepts on the wire. `platform`, `chat_id`, `chat_type`,
|
||||
`user_id`, `user_name`, `thread_id`, `chat_name`, and `chat_topic` are always
|
||||
present (may be `null`); the rest are included only when set.
|
||||
|
||||
| Field | Type | Always sent | Meaning |
|
||||
| --- | --- | --- | --- |
|
||||
| `platform` | string | yes | Platform name (matches the descriptor's `platform`). |
|
||||
| `chat_id` | string | yes | Primary conversation id (channel/chat). Session-key discriminator. |
|
||||
| `chat_type` | string | yes | `dm` / `group` / `channel` / `thread` / `forum`. |
|
||||
| `chat_name` | string\|null | yes | Human-readable chat name. |
|
||||
| `user_id` | string\|null | yes | Message author id. Session-key discriminator. |
|
||||
| `user_name` | string\|null | yes | Author display name. |
|
||||
| `thread_id` | string\|null | yes | Thread/forum-topic id when in a thread. Session-key discriminator. |
|
||||
| `chat_topic` | string\|null | yes | Channel topic/description (Discord, Slack). |
|
||||
| `user_id_alt` | string | no | Platform-specific stable alt id (Signal UUID, Feishu union_id). |
|
||||
| `chat_id_alt` | string | no | Alternate chat id (e.g. Signal group internal id). |
|
||||
| `scope_id` | string | no | Platform-neutral **scope** discriminator: Discord guild / Slack workspace / Matrix server. **REQUIRED for Discord/Slack scope isolation.** Session-key discriminator. (Canonical name as of the D-Q2.5 wire migration.) |
|
||||
| `guild_id` | string | no | **Legacy alias, no longer read by the connector.** As of D-Q2.5c the connector reads and writes only `scope_id`; the gateway's agent-wide `SessionSource.to_dict()` still emits `guild_id` (mirrored to `scope_id`) for non-relay session persistence, so it may still appear on the wire but the connector ignores it. Do not depend on it. |
|
||||
| `parent_chat_id` | string | no | Parent channel when `chat_id` refers to a thread. |
|
||||
| `message_id` | string | no | Id of the triggering message (for pin/reply/react). |
|
||||
|
||||
> `is_bot` (author-is-a-bot/webhook classification) exists on the gateway-side
|
||||
> dataclass but is **intentionally NOT on the wire** in v1 — it is not part of
|
||||
> `to_dict()`. Do not add it to the connector's `SessionSource` until it is
|
||||
> first added here and to `to_dict()` (additive bump).
|
||||
|
||||
### SessionSource discriminators per platform
|
||||
|
||||
| Platform | chat_id | chat_type | user_id | thread_id | scope_id |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| **Discord** | channel id | `dm`/`group`/`thread` | author id | thread channel id (threads) | **guild id** (REQUIRED for server isolation) |
|
||||
| **Telegram** | chat id | `dm`/`group`/`forum` | from id | forum topic id (forums) | — |
|
||||
|
||||
**Get Discord's `guild_id` wrong and two servers collide into one session.**
|
||||
This is the #1 High-severity risk. The gateway's `build_session_key()` is the
|
||||
conformance oracle: for a given `SessionSource`, the connector's normalization
|
||||
must produce the same key the Python adapter would. (The Phase-1 stub tests
|
||||
assert known-input → known-key.)
|
||||
|
||||
### Bot identity vs tenant (single-bot consolidation, Appendix A)
|
||||
|
||||
The envelope carries the **originating bot identity** as a field **distinct from
|
||||
tenant**. Tenant is resolved from the event's own discriminator (Discord
|
||||
`guild_id`, Telegram `chat_id`, webhook path/subdomain) — **never** from which
|
||||
token/socket/process delivered it. This keeps one shared bot able to front many
|
||||
tenants (Phase 6) without overloading an existing field.
|
||||
|
||||
### Author-first resolution + the account-link (DM) path (Phase 7)
|
||||
|
||||
Phase 7 adds **self-serve, per-user onboarding to a shared bot**, which changes
|
||||
*which* discriminator resolves the instance for a routed inbound message — and
|
||||
adds a management path for users to bind their own account.
|
||||
|
||||
**Author-first resolution (the multi-tenant-guild rule, D-7.2).** A single
|
||||
Discord guild may hold **many** tenants — different members each linked to their
|
||||
own agent. So for delivery the connector resolves the destination instance from
|
||||
the **authenticated author binding** (`user_instance_binding`, keyed by
|
||||
`(tenant, platform, platform_user_id)` via `resolveByUser`), **NOT** by a
|
||||
guild→instance route. Concretely:
|
||||
|
||||
- A routed message authored by a **linked** user reaches **only that user's**
|
||||
instance — even when a second linked user in the **same guild** is served by a
|
||||
different instance (each reaches only their own).
|
||||
- A message authored by an **unlinked** user resolves to **no** instance and is
|
||||
dropped (**fail-closed** — never broadcast to the guild's other tenants).
|
||||
- The author id used is the **authentic `user_id` off the observed event**, the
|
||||
same `SessionSource.user_id` documented above — never a value asserted by a
|
||||
gateway or carried in a management frame.
|
||||
|
||||
This is the per-`user_id` owner-only routing the connector enforces in
|
||||
`WsGatewayDelivery` (the gateway-side multi-tenant-guild E2E driver
|
||||
`gateway_multitenant_guild_driver.py` is the cross-repo oracle).
|
||||
|
||||
**The account-link (DM) path.** A user binds their account to an instance with a
|
||||
one-time code, redeemed by DMing the shared bot:
|
||||
|
||||
1. The owner triggers a link from the Portal (or a self-hosted CLI). The
|
||||
connector mints a short-lived **link code** for the **authenticated**
|
||||
instance (`POST /manage/link`; instanceId comes from the caller's principal —
|
||||
a NAS-signed `aud=agent:{instanceId}` token or the instance's own per-gateway
|
||||
secret — **never** the request body).
|
||||
2. The user sends `/link <code>` as a **direct message** to the shared bot from
|
||||
the account they want to bind.
|
||||
3. The connector's inbound observer **consumes** that DM (it is not routed to any
|
||||
agent) and writes the `user_instance_binding` using the **authentic
|
||||
`user_id`** off the observed DM event. From then on, author-first resolution
|
||||
routes that user's messages to the bound instance.
|
||||
|
||||
**Opt-out is connector-authoritative.** Deprovisioning an instance
|
||||
(`POST /manage/deprovision`) drops its author bindings (so its users stop
|
||||
resolving to it) **and** revokes its per-gateway secret (so its socket can no
|
||||
longer authenticate — the next WS upgrade is closed **4401**). A gateway that
|
||||
sees a **4401 close after a previously-successful handshake** treats it as a
|
||||
terminal revocation: it stops reconnecting and reports the relay platform as
|
||||
**disabled** (not a retryable error). A 4401 *before* any successful handshake
|
||||
stays retryable (a cold-start / not-yet-provisioned race, not a revocation).
|
||||
|
||||
### 3.2 Going-idle / buffered-flip primitive (§5.3)
|
||||
|
||||
A scale-to-zero PRIMITIVE (not the behaviour — nothing here decides to sleep or
|
||||
suspends a machine; a later workstream consumes these frames). It lets a gateway
|
||||
enter a drain/idle transition without losing inbound that arrives while it is
|
||||
gone, by making the connector buffer for that instance and replay on reconnect.
|
||||
|
||||
Three frames (all keyed by the connection's **authenticated** per-instance id —
|
||||
read off the stored secret record at the WS upgrade, never asserted in a frame):
|
||||
|
||||
- `{"type":"going_idle"}` (gateway → connector) — emitted as part of the
|
||||
gateway's EXISTING drain transition (the adapter sends it before tearing down
|
||||
the socket). Asks the connector to flip this instance to **buffered-only**.
|
||||
- `{"type":"going_idle_ack"}` (connector → gateway) — the connector has flipped:
|
||||
live delivery has stopped and subsequent inbound for this instance buffers
|
||||
durably. The gateway **stays serving until this ack** (so an event landing in
|
||||
the flip window is delivered live, not lost — the same SUBSCRIBE-before-serve
|
||||
ordering discipline as the bus). Only after the ack is it safe to close.
|
||||
- `{"type":"inbound_ack", "bufferId"}` (gateway → connector) — durable receipt of
|
||||
a buffered `inbound` delivery (which carries its `bufferId`) replayed on
|
||||
reconnect. The connector acks the buffer entry only after this, giving
|
||||
drain-without-dup on the **delivery leg**: an instance that dies mid-drain
|
||||
redelivers exactly the unacked tail; an acked entry never redelivers.
|
||||
|
||||
**Buffer + drain.** While flipped, the connector appends inbound to a durable
|
||||
per-instance delivery-leg buffer (`delivery:<instanceId>`) instead of pushing it
|
||||
live. On the gateway's **reconnect** (a NET-NEW reconnect loop re-dials +
|
||||
re-handshakes after an unexpected close), the new handshake triggers the
|
||||
connector to drain that backlog over the new socket **in order, ack-gated**,
|
||||
then clear the flip so live delivery resumes. This reuses the same
|
||||
`drainWithoutDup` machinery as the Discord→connector ingest leg, applied to the
|
||||
connector→gateway delivery leg. Connector-authoritative throughout: a gateway can
|
||||
only flip/drain ITS OWN instance.
|
||||
|
||||
> NOT in scope (deferred behaviour): the autonomous idle timer that DECIDES to
|
||||
> drain, the actual machine suspend, and the NAS suspended-health model. The
|
||||
> primitive is "when the gateway drains, relay flips to buffered + replays on
|
||||
> reconnect, with no loss/dup"; WHAT triggers the drain is out of scope.
|
||||
|
||||
### 3.3 Wake poke (§5.2)
|
||||
|
||||
The other half of the sleep/wake loop: how a SUSPENDED gateway finds out it has
|
||||
buffered work waiting. A PRIMITIVE — nothing here suspends a machine; it wires
|
||||
the wake SIGNAL so a future scale-to-zero behaviour layer can rely on "buffered
|
||||
⇒ wake poked."
|
||||
|
||||
- **Registration.** The gateway registers a **wake URL** at enroll/provision —
|
||||
any reachable URL the connector can GET to wake it (a Fly autostart hostname,
|
||||
a dashboard host). Self-hosted: `hermes gateway enroll --wake-url <url>` (or
|
||||
`GATEWAY_RELAY_WAKE_URL` / `gateway.relay_wake_url`). Managed/NAS: stamped into
|
||||
the container env beside `GATEWAY_RELAY_URL`. Forwarded in the
|
||||
`/relay/provision` body as `wakeUrl` and stored per-instance on the connector's
|
||||
secret record (gateway-asserted but safely scoped — same posture as
|
||||
`instanceId`; the org/tenant stays token-verified, so a gateway can only
|
||||
register a wake target for ITS OWN instance). DISTINCT from the retired
|
||||
`gatewayEndpoint`: a **poke target**, not a delivery target.
|
||||
- **The poke.** When a buffered-only (going-idle) destination receives its FIRST
|
||||
buffered event, the connector issues a **payload-free, unsigned GET** to that
|
||||
instance's registered `wakeUrl`, **directly** (NOT NAS-mediated — relay stays
|
||||
NAS-independent). It carries no tenant data and no inbound: it only says "you
|
||||
have buffered work, reconnect." Tenant authority is re-established the normal
|
||||
way when the gateway re-dials (the authenticated WS upgrade), so a leaked/
|
||||
guessed wake URL can at worst cause a spurious reconnect of ITS OWN instance.
|
||||
Rate-limited per instance (one poke per cooldown window, not per event), and
|
||||
best-effort — a failed poke is swallowed; the gateway still drains whenever it
|
||||
next reconnects on its own. No new frame: the wake is an out-of-band HTTP GET,
|
||||
not a relay-WS message (the socket is down — that's the whole point).
|
||||
|
||||
> NOT in scope (deferred behaviour): the actual machine suspend (Fly
|
||||
> `autostop:"suspend"`) and the autonomous idle timer that decides to sleep. The
|
||||
> primitive is "buffered event for a sleeping instance ⇒ its wakeUrl gets poked";
|
||||
> WHAT makes the instance sleep (and wake-to-serve) is the behaviour layer.
|
||||
|
||||
### 3.4 Obligations on a future scale-to-zero behaviour layer
|
||||
|
||||
§3.2 and §3.3 ship the **primitives**; this section is the **contract a separate
|
||||
scale-to-zero behaviour workstream must honour to consume them safely.** It owns
|
||||
the *decision* to suspend, the actual machine suspend, and the platform/health
|
||||
model — none of which live here — but it MUST hold these guarantees, which the
|
||||
primitives assume:
|
||||
|
||||
1. **Register a `wakeUrl` before the instance can ever be suspended.** A
|
||||
suspended instance with no registered `wakeUrl` is a black hole — buffered
|
||||
inbound never triggers a poke, so it sleeps through its own traffic until
|
||||
something else reconnects it. The behaviour layer MUST ensure a reachable
|
||||
wake target is registered (self-hosted: `--wake-url`; managed: stamped) as a
|
||||
precondition of allowing suspend. A wake URL that is unreachable while the
|
||||
machine is suspended (e.g. points at the suspended machine itself with no
|
||||
platform autostart in front) is equivalent to none.
|
||||
2. **Drain through `going_idle` → await `going_idle_ack` BEFORE tearing down the
|
||||
socket or suspending.** Never suspend with an un-acked flip in flight. The
|
||||
ack is the connector's confirmation that delivery for this instance is now
|
||||
buffered-only; a machine that suspends after sending `going_idle` but before
|
||||
the ack can drop the inbound that races the flip. The gateway already gates
|
||||
socket teardown on the ack (Q-5.3c); the suspend step MUST sit *after* a
|
||||
clean drain completes, not race it.
|
||||
3. **Keep the NET-NEW reconnect loop live as a precondition of suspend.** The
|
||||
wake→drain contract is "poke ⇒ the gateway re-dials ⇒ the connector drains on
|
||||
the reconnect handshake." If the reconnect loop is disabled, a poke lands on a
|
||||
machine that never re-dials and the buffer strands. The behaviour layer must
|
||||
not suspend an instance whose relay transport won't reconnect on wake.
|
||||
4. **Treat suspended ≠ down in the health model (Q-5.3b).** A suspended instance
|
||||
is healthy-asleep, not failed. The health/monitoring layer MUST distinguish
|
||||
the two (e.g. via the platform machine-state) so a suspended instance is not
|
||||
restarted, alerted on, or reaped as unhealthy — that would defeat the suspend
|
||||
and can race the wake/drain.
|
||||
5. **The wake poke is best-effort and rate-limited — do not assume exactly-once
|
||||
or immediate wake.** At most one poke per cooldown window per instance, and a
|
||||
failed poke is swallowed. The behaviour layer must not rely on the poke as a
|
||||
guaranteed/prompt signal; correctness still rests on "the gateway drains
|
||||
whenever it next reconnects." A belt-and-suspenders wake (e.g. a scheduled
|
||||
job that also reconnects) is the behaviour layer's call, not the primitive's.
|
||||
6. **Suspend only when genuinely idle — and idle is connector-observable, not
|
||||
gateway-guessed.** WHAT counts as idle (no in-flight turn + no inbound for N
|
||||
min) is the behaviour layer's policy, but it must compose with the existing
|
||||
drain machinery (`gateway_state` running→draining) rather than introduce a
|
||||
parallel relay-only idle path — the same integration constraint §3.2 places
|
||||
on `going_idle`.
|
||||
|
||||
These are guarantees the behaviour layer OWES the primitives; the primitives owe
|
||||
the behaviour layer only what §3.2/§3.3 already specify (a flip-on-going_idle,
|
||||
a durable per-instance buffer + ack-gated reconnect drain, and a poke on the
|
||||
first buffered event for a flipped instance).
|
||||
|
||||
---
|
||||
|
||||
## 4. Outbound: action set
|
||||
|
||||
The gateway calls the transport with action dicts. Source of truth:
|
||||
`gateway/relay/transport.py` + `gateway/relay/adapter.py`.
|
||||
|
||||
| `op` | Fields | Result |
|
||||
| --- | --- | --- |
|
||||
| `send` | `chat_id`, `content`, `reply_to?`, `metadata?` | `{success: bool, message_id?, error?}` |
|
||||
| `edit` | `chat_id`, `message_id`, `content`, `metadata?` | `{success: bool, error?}` |
|
||||
| `typing` | `chat_id` | `{success: bool}` |
|
||||
| `follow_up` | `session_key`, `kind`, `content`, `metadata?` | `{success: bool, message_id?, error?}` |
|
||||
|
||||
`get_chat_info(chat_id)` is a separate proxied call returning at least
|
||||
`{name, type}`. Media actions follow the same envelope shape (deferred to a
|
||||
later contract revision; additive).
|
||||
|
||||
**`follow_up` (A2 capability action).** Some inbound payloads carry a credential
|
||||
that acts on the **shared** bot identity (e.g. a Discord interaction follow-up
|
||||
token). Per §6 the connector strips that at the edge and binds it in its
|
||||
capability vault keyed by the session; it **never reaches the gateway**. To use
|
||||
it, the gateway issues `follow_up` naming the **session it is already in**
|
||||
(`session_key`) plus the capability `kind` (e.g. `discord.interaction_token`) —
|
||||
**never a token**. The connector resolves the real value from its vault,
|
||||
enforces the tenant match (tenant B can never wield tenant A's capability), and
|
||||
egresses. `success: false` when the capability is absent/expired or the tenant
|
||||
doesn't match — the gateway has nothing to retry with, by design (a leaked
|
||||
gateway holds zero capability material). Source of truth:
|
||||
`gateway/relay/transport.py` (`send_follow_up`) + `gateway/relay/adapter.py`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Interrupt (`/stop`) routing
|
||||
|
||||
- **Gateway → connector:** `send_interrupt(session_key, reason?)` egresses a
|
||||
mid-turn `/stop` over the outbound WS. The connector MUST forward it to the
|
||||
gateway instance running that `session_key` (the routing invariant).
|
||||
- **Connector → gateway:** an inbound interrupt for a `session_key` is delivered
|
||||
as an `interrupt_inbound` frame down the gateway's outbound WS (§3 transport
|
||||
note) — routed cross-instance via the relay bus to whichever instance holds
|
||||
the socket — and bridged by the adapter's `on_interrupt(session_key, chat_id)`
|
||||
into the existing per-session interrupt mechanism, cancelling exactly that turn
|
||||
(siblings untouched).
|
||||
|
||||
Both directions ride the gateway's outbound WS: the gateway→connector `/stop`
|
||||
egresses over it, and the connector→gateway interrupt rides the same `inbound`
|
||||
back-channel as a normalized event.
|
||||
|
||||
---
|
||||
|
||||
## 6. Trust boundary & signed-body handling (A2)
|
||||
|
||||
**The connector is the sole crypto/identity boundary. The gateway re-validates
|
||||
nothing.**
|
||||
|
||||
Webhook signatures (Discord ed25519, Twilio HMAC, WeCom BizMsgCrypt) are
|
||||
computed over exact raw bytes, and some payloads are *encrypted* with a shared
|
||||
secret. The connector fronts a **shared** bot for many tenants and holds every
|
||||
tenant's platform secrets, so it:
|
||||
|
||||
- **verifies / decrypts at the edge** (the only place the secrets live),
|
||||
- **normalizes** the payload into a tenant-scoped `MessageEvent` (§3),
|
||||
- **strips any shared-identity capability** out of the payload and binds it in
|
||||
its capability vault, keyed by the session (see §4 `follow_up`),
|
||||
- **forwards only the sanitized `MessageEvent`** — never the raw signed body.
|
||||
|
||||
The gateway therefore performs **no** platform signature/crypto verification on
|
||||
the relay path; it trusts the normalized event. This is an enforced invariant on
|
||||
the gateway side (`tests/gateway/relay/test_relay_sheds_crypto.py`: the relay
|
||||
package imports/calls no platform-crypto).
|
||||
|
||||
**Why not "forward the signed body byte-for-byte so the gateway re-validates"?**
|
||||
That earlier model is incoherent under an untrusted, disposable tenant gateway:
|
||||
|
||||
- Re-validating Twilio HMAC / WeCom crypto would require handing the gateway the
|
||||
**shared signing secret** — which is itself the leak, and on a shared bot it's
|
||||
a *cross-tenant* leak.
|
||||
- WeCom payloads are encrypted with the shared secret; the connector must decrypt
|
||||
at the edge just to route, so forwarding ciphertext would again require giving
|
||||
the gateway the secret.
|
||||
- A Discord interaction token lives **inside** the signed JSON body — you cannot
|
||||
both preserve the bytes and strip the credential; they are the same bytes.
|
||||
|
||||
So byte-preservation is abandoned deliberately: the connector re-serializes the
|
||||
sanitized event and the gateway trusts it. This also unifies the passthrough and
|
||||
relay planes — both are "verify at the edge → emit a normalized event," differing
|
||||
only in transport. See `docs/capability-trust-boundary.md` (connector repo:
|
||||
`gateway-gateway`) for the full A2 rationale and the connector-side vault.
|
||||
|
||||
### 6.1 Channel authentication (the connector⇄gateway link itself)
|
||||
|
||||
A2 makes the connector the sole holder of platform secrets while the gateway may
|
||||
be **customer-managed and internet-exposed**, so the connector⇄gateway channel
|
||||
is itself authenticated. The gateway holds an enrollment- or provision-issued
|
||||
**per-gateway secret** (`hermes gateway enroll` → connector `/relay/enroll`, or
|
||||
managed self-provision → `/relay/provision`) that authenticates its outbound WS
|
||||
upgrade. It is an HMAC-SHA256 scheme with a multi-secret rotation verify list
|
||||
(gateway side: `gateway/relay/auth.py`; connector side:
|
||||
`src/core/relayAuthToken.ts`).
|
||||
|
||||
| Leg | Credential | Mechanism |
|
||||
|-----|-----------|-----------|
|
||||
| Gateway → connector WS upgrade | per-gateway secret | An `Authorization` bearer header on the `/relay` upgrade. The token is `base64url(payload:exp:sig)` where `payload = gatewayId` and `sig = HMAC(payload:exp, secret)`. Connector verifies and rejects the upgrade (**close 4401**) on mismatch/absence/revocation. The authenticated tenant comes from the connector's store, never the `hello` frame. |
|
||||
| Connector → gateway inbound (`inbound` / `interrupt_inbound` frames) | — (rides the authenticated WS) | Inbound is pushed down the gateway's already-authenticated outbound socket (§3), so no per-message signature is needed. A **per-tenant delivery key** is still issued at enroll/provision and retained for forward-compat, but is no longer used to sign inbound. |
|
||||
|
||||
This is the **channel** authenticator — distinct from platform crypto, which the
|
||||
relay path still sheds entirely (§6). The gateway holds zero platform secrets;
|
||||
the per-gateway secret authenticates only the connector link. Full threat model +
|
||||
enrollment/rotation/kill-switch design: `docs/connector-gateway-auth-design.md`
|
||||
(connector repo).
|
||||
|
||||
---
|
||||
|
||||
## 7. Per-instance delivery & the management plane (Phase 6)
|
||||
|
||||
Phases 1–5 treat the connector as a single-tenant front: inbound events for a
|
||||
tenant fan out to that tenant's gateway socket(s). **Phase 6 makes delivery
|
||||
per-INSTANCE** — a shared bot can front many users/agents in one tenant (one
|
||||
Discord guild, one Telegram bot) without cross-delivery — and adds a small
|
||||
**management plane** the agent (or a managed Portal) uses to declare who-sees-what
|
||||
and what's-relevant. All of this lives **connector-side**; the gateway's only new
|
||||
responsibility is to **declare its relevance policy** at boot (§7.3).
|
||||
|
||||
### 7.1 The delivery gate (connector-side, informational)
|
||||
|
||||
For each inbound event the connector decides which instances receive it by
|
||||
composing three AND-ed filters. The gateway does not implement these — they run
|
||||
in the connector — but they define the delivery semantics the gateway relies on:
|
||||
|
||||
| Layer | Question | Source of truth |
|
||||
| --- | --- | --- |
|
||||
| **owner / scope ∧ principal** | May this instance *see* this author here? | per-user `user_id → instance` bindings (the owner floor) + per-instance `(guild, channel)` scope grants + an `owner-only` / `allow-list` / `any` principal policy. |
|
||||
| **visibility floor** | Can the instance's bound owner actually `VIEW_CHANNEL` this in Discord? | live Discord ACL (effective permissions), fail-closed. Narrows an over-broad scope grant downward. |
|
||||
| **relevance** | *Given* it may see it, should the agent engage? | the relevance policy declared in §7.3 (address-gating / free-response / allow-bots). |
|
||||
|
||||
The composition only ever **narrows** delivery (`deliver ⇔ authorized ∧ visible
|
||||
∧ relevant`); the **owner floor bypasses the relevance layer** (an author's own
|
||||
message always reaches their own instance — you don't @mention your own agent).
|
||||
A message authored by an unbound user reaches no instance (fail-closed). The
|
||||
full design + invariants live in the connector repo
|
||||
(`NousResearch/gateway-gateway`); this section is the gateway-facing summary.
|
||||
|
||||
### 7.2 Management routes (connector-side, authenticated)
|
||||
|
||||
The connector mounts authenticated management routes. They share the **same
|
||||
dual-auth** as the WS upgrade: either a managed NAS-signed `aud=agent:{instanceId}`
|
||||
RS256 JWT, **or** the gateway's own per-gateway secret bearer (§6.1
|
||||
`make_upgrade_token`). In both cases the connector resolves the authoritative
|
||||
`{tenant, instanceId}` from its **stored** record — **never** from the request
|
||||
body (a body-asserted `instanceId` is ignored).
|
||||
|
||||
| Route | Purpose |
|
||||
| --- | --- |
|
||||
| `POST /manage/link` | Issue a short-lived code to bind a platform account to the authenticated instance (the `/link <code>` flow; the connector reads the authentic `user_id` off the inbound event). |
|
||||
| `POST /manage/scope`, `/manage/scope/release` | Claim / release a `(guild, channel)` scope for the authenticated instance. A channel is owned by at most one instance (non-overlap is a PK constraint). |
|
||||
| `POST /manage/principal` | Set the instance's principal policy (`owner-only` \| `allow-list` \| `any`). |
|
||||
| `POST /manage/dm-default` | Set the user's DM-default instance (DM tie-break when a user linked more than one). |
|
||||
| `POST /relay/policy` | Declare the instance's **relevance policy** (§7.3). |
|
||||
|
||||
These are connector-owned (the management plane is not part of the gateway's
|
||||
agent path); the gateway only calls `POST /relay/policy` (§7.3). The others are
|
||||
driven by the managed Portal / `hermes` CLI.
|
||||
|
||||
### 7.3 Relevance-policy declaration (the gateway's responsibility)
|
||||
|
||||
The relevance layer (§7.1) is the per-tenant parity for the gateway's own
|
||||
behaviour knobs (`require_mention`, `free_response_channels`,
|
||||
`{PLATFORM}_ALLOW_BOTS`). So the **same** behaviour governs relay delivery, the
|
||||
gateway projects those knobs into a **platform-agnostic** policy and POSTs it to
|
||||
`POST /relay/policy` at boot (after its per-gateway secret is resolved).
|
||||
|
||||
Body (`gateway/relay/__init__.py` `relay_relevance_policy()` → `send_relay_policy()`):
|
||||
|
||||
| Field | Type | Projected from | Meaning |
|
||||
| --- | --- | --- | --- |
|
||||
| `platform` | string | the fronted platform (`relay_platform_identity`) | which platform this policy applies to. |
|
||||
| `requireAddress` | bool | `require_mention` | a non-owner message must @mention / reply-to the bot to be relevant. |
|
||||
| `freeResponseScopes` | string[] | `free_response_channels` | scope (channel) ids where `requireAddress` is waived. Same scope vocabulary as §7.1's scope grants. |
|
||||
| `allowOtherBots` | bool | `{PLATFORM}_ALLOW_BOTS ∈ {mentions, all}` | admit bot-authored messages (default off). |
|
||||
|
||||
Auth is the per-gateway upgrade token (§6.1), so the connector attaches the
|
||||
policy to the authenticated instance. The gateway is the **source of truth** and
|
||||
re-declares **every boot** (a full replace, mirroring the `routeKeys` upsert at
|
||||
provision — self-healing). When the projected policy is all-default the gateway
|
||||
sends nothing (the connector's absent-row default already matches). The POST is
|
||||
**fail-soft**: a failure logs and boot proceeds — relevance is an optimization
|
||||
layered on the authorization gate (§7.1), never a boot dependency. There is **no
|
||||
new gateway inbound surface** and **no new credential** — it reuses the
|
||||
per-gateway secret and the same host as `/relay/provision`.
|
||||
|
||||
> A relevance drop happens **before** the connector wakes a scaled-to-zero agent
|
||||
> (Phase 5), so excluded chatter never spins an agent up — relevance is the
|
||||
> primary scale-to-zero lever as well as a correctness filter.
|
||||
|
||||
---
|
||||
|
||||
## 8. Versioning policy
|
||||
|
||||
- `contract_version` is an int; bump **only** for additive changes during the
|
||||
experimental phase (new optional fields, new `op`s).
|
||||
- A breaking change (renamed/removed field, changed semantics) requires a
|
||||
coordinated update of both repos and a version bump.
|
||||
- The connector's first PR references the commit SHA of this file it implements
|
||||
against.
|
||||
@@ -0,0 +1,195 @@
|
||||
# Network Egress Isolation for Docker Deployments
|
||||
|
||||
When running Hermes inside Docker, the default `network_mode: host` gives the
|
||||
agent process unrestricted outbound network access. This guide shows how to
|
||||
segment traffic so the agent core can only reach the services it needs, while
|
||||
blocking arbitrary outbound connections.
|
||||
|
||||
This is primarily a defense against prompt injection attacks that attempt to
|
||||
exfiltrate data via `curl`, `wget`, or raw HTTP from tool-generated shell
|
||||
commands.
|
||||
|
||||
## Threat Model
|
||||
|
||||
The Hermes [SECURITY.md](../../SECURITY.md) §2 defines the trust model. The
|
||||
terminal backend is the primary execution boundary. However, when running with
|
||||
`network_mode: host`, any command the agent executes can reach any endpoint on
|
||||
the network, including external ones.
|
||||
|
||||
Network egress isolation adds a second layer: even if a malicious command
|
||||
executes inside the container, it cannot reach endpoints outside the
|
||||
explicitly allowlisted set.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ Docker Network: internal (no internet) │
|
||||
│ │
|
||||
│ ┌──────────────┐ ┌──────────────────┐ │
|
||||
│ │ hermes-agent │ │ hermes-dashboard │ │
|
||||
│ └──────┬───────┘ └────────┬─────────┘ │
|
||||
│ │ │ │
|
||||
│ ▼ │ │
|
||||
│ ┌──────────────┐ │ │
|
||||
│ │ hermes-gtw │◄───────────┘ │
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
└──────────┼───────────────────────────────────┘
|
||||
│
|
||||
┌──────────┼───────────────────────────────────┐
|
||||
│ Docker Network: egress (internet-capable) │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ ┌─────────────────┐ │
|
||||
│ │ egress-proxy │──► allowlisted hosts │
|
||||
│ │ (squid / envoy) │ │
|
||||
│ └─────────────────┘ │
|
||||
└──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Two Docker networks:
|
||||
|
||||
- **`internal`** — no default route, no internet access. The agent, dashboard,
|
||||
and gateway run here.
|
||||
- **`egress`** — has internet access. Only services that need to reach external
|
||||
APIs are attached to this network.
|
||||
|
||||
The gateway service is dual-homed (attached to both networks) so it can
|
||||
receive inbound messages from Telegram/Slack/etc. and forward them to the
|
||||
agent on the internal network.
|
||||
|
||||
## Compose Configuration
|
||||
|
||||
Override the default `docker-compose.yml` with a
|
||||
`docker-compose.override.yml`:
|
||||
|
||||
```yaml
|
||||
# docker-compose.override.yml
|
||||
# Network egress isolation for production deployments.
|
||||
#
|
||||
# Usage:
|
||||
# HERMES_UID=$(id -u) HERMES_GID=$(id -g) docker compose up -d
|
||||
#
|
||||
# This overrides network_mode: host with isolated Docker networks.
|
||||
|
||||
networks:
|
||||
internal:
|
||||
driver: bridge
|
||||
internal: true # no default route, no internet
|
||||
egress:
|
||||
driver: bridge
|
||||
|
||||
services:
|
||||
gateway:
|
||||
network_mode: "" # clear the host-mode default
|
||||
networks:
|
||||
- internal
|
||||
- egress # needs outbound for Telegram, LLM APIs
|
||||
ports:
|
||||
- "127.0.0.1:9119:9119" # dashboard proxy, localhost only
|
||||
|
||||
dashboard:
|
||||
network_mode: ""
|
||||
networks:
|
||||
- internal # internal only, no egress needed
|
||||
```
|
||||
|
||||
### With an Egress Proxy (Recommended)
|
||||
|
||||
For tighter control, route all outbound traffic through an HTTP proxy with
|
||||
an explicit allowlist:
|
||||
|
||||
```yaml
|
||||
# docker-compose.override.yml (with egress proxy)
|
||||
|
||||
networks:
|
||||
internal:
|
||||
driver: bridge
|
||||
internal: true
|
||||
egress:
|
||||
driver: bridge
|
||||
|
||||
services:
|
||||
gateway:
|
||||
network_mode: ""
|
||||
networks:
|
||||
- internal
|
||||
- egress
|
||||
environment:
|
||||
- HTTP_PROXY=http://egress-proxy:3128
|
||||
- HTTPS_PROXY=http://egress-proxy:3128
|
||||
- NO_PROXY=hermes,hermes-dashboard,localhost
|
||||
|
||||
dashboard:
|
||||
network_mode: ""
|
||||
networks:
|
||||
- internal
|
||||
|
||||
egress-proxy:
|
||||
image: ubuntu/squid:6.10-24.04_edge
|
||||
networks:
|
||||
- egress
|
||||
volumes:
|
||||
- ./config/squid-allowlist.conf:/etc/squid/conf.d/allowlist.conf:ro
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
Example `config/squid-allowlist.conf`:
|
||||
|
||||
```
|
||||
# Only allow HTTPS CONNECT to these hosts
|
||||
acl allowed_hosts dstdomain api.openai.com
|
||||
acl allowed_hosts dstdomain api.anthropic.com
|
||||
acl allowed_hosts dstdomain openrouter.ai
|
||||
acl allowed_hosts dstdomain generativelanguage.googleapis.com
|
||||
acl allowed_hosts dstdomain api.telegram.org
|
||||
acl allowed_hosts dstdomain api.github.com
|
||||
acl allowed_hosts dstdomain discord.com
|
||||
|
||||
http_access allow CONNECT allowed_hosts
|
||||
http_access deny all
|
||||
```
|
||||
|
||||
Adjust the allowlist to match your LLM provider and messaging platform.
|
||||
|
||||
## Validating the Setup
|
||||
|
||||
After bringing up the stack, verify isolation:
|
||||
|
||||
```bash
|
||||
# From the agent container: this should FAIL (no egress)
|
||||
docker compose exec gateway \
|
||||
curl -sf --max-time 5 https://example.com && echo "FAIL: egress not blocked" || echo "OK: egress blocked"
|
||||
|
||||
# From the agent container: this should SUCCEED (internal network)
|
||||
docker compose exec gateway \
|
||||
curl -sf --max-time 5 http://hermes-dashboard:9119/health && echo "OK: internal reachable" || echo "FAIL"
|
||||
|
||||
# If using egress proxy: this should SUCCEED (allowlisted)
|
||||
docker compose exec gateway \
|
||||
curl -sf --max-time 5 --proxy http://egress-proxy:3128 https://api.openai.com/v1/models && echo "OK" || echo "FAIL"
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- **DNS resolution:** The `internal` network can still resolve external DNS
|
||||
names unless you also run a local DNS resolver that blocks external queries.
|
||||
For most threat models this is acceptable since DNS resolution alone does not
|
||||
exfiltrate meaningful data.
|
||||
|
||||
- **Not a substitute for sandbox backends:** This guide isolates the agent
|
||||
*container's* network. If you use the default local terminal backend, tool
|
||||
commands execute inside the same container. For stronger isolation, combine
|
||||
network segmentation with a sandboxed terminal backend (Docker, Modal,
|
||||
Daytona).
|
||||
|
||||
- **Platform adapters need egress:** The gateway service needs outbound access
|
||||
to reach messaging platform APIs. If you add new platform adapters, add their
|
||||
API endpoints to the proxy allowlist.
|
||||
|
||||
## Related
|
||||
|
||||
- [SECURITY.md](../../SECURITY.md) — Hermes trust model and vulnerability reporting
|
||||
- [Terminal backends](../../README.md) — sandboxed execution targets
|
||||
- [docker-compose.yml](../../docker-compose.yml) — default compose configuration
|
||||
@@ -0,0 +1,631 @@
|
||||
# Session Lifecycle
|
||||
|
||||
> **Audience:** Gateway developers and maintainers
|
||||
> **Source files:** `gateway/session.py` (~1444 lines), `gateway/run.py` (~16800 lines), `gateway/config.py`
|
||||
> **Last updated:** 2026-06-16
|
||||
|
||||
## Overview
|
||||
|
||||
A **session** represents a continuous conversation between the agent and one or more users on a
|
||||
messaging platform. The session lifecycle governs when conversations persist, when they reset,
|
||||
how they survive gateway restarts, and how messages queue during concurrent operations.
|
||||
|
||||
The session system lives primarily in two modules:
|
||||
|
||||
- `gateway/session.py` — Data model (`SessionSource`, `SessionEntry`, `SessionContext`),
|
||||
key generation (`build_session_key`), and the main store (`SessionStore`).
|
||||
- `gateway/run.py` — Gateway runner (`GatewayRunner`) that wires sessions into the message
|
||||
processing pipeline: session expiry watching, agent caching, restart recovery, and message
|
||||
queuing.
|
||||
|
||||
---
|
||||
|
||||
## 1. SessionSource — Message Origin Descriptor
|
||||
|
||||
`SessionSource` is a frozen record of *where a message came from*. It is attached to every
|
||||
incoming `MessageEvent` and used for routing, isolation, and context injection.
|
||||
|
||||
### Fields
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `platform` | `Platform` | *(required)* | Enum identifying the messaging platform (telegram, discord, slack, signal, whatsapp, matrix, local, etc.). |
|
||||
| `chat_id` | `str` | *(required)* | Platform-level chat/group/channel identifier. Routed through the adapter's `chat_id_key` transform. |
|
||||
| `chat_name` | `Optional[str]` | `None` | Human-readable name of the chat or group. |
|
||||
| `chat_type` | `str` | `"dm"` | One of `"dm"`, `"group"`, `"channel"`, `"thread"`. Controls session key generation and isolation. |
|
||||
| `user_id` | `Optional[str]` | `None` | Platform-specific user identifier. Used for authorization and per-user session isolation. |
|
||||
| `user_name` | `Optional[str]` | `None` | Display name of the message author. Injected into system prompt. |
|
||||
| `thread_id` | `Optional[str]` | `None` | Forum topic / Discord thread / Slack thread identifier. Differentiates threaded conversations. |
|
||||
| `chat_topic` | `Optional[str]` | `None` | Channel topic or description (Discord channel topic, Slack channel purpose). |
|
||||
| `user_id_alt` | `Optional[str]` | `None` | Platform-specific stable alternative ID (Signal UUID, Feishu union_id). Used when `user_id` is ephemeral. |
|
||||
| `chat_id_alt` | `Optional[str]` | `None` | Signal group internal ID — maps a Signal group V2 identifier to its canonical form. |
|
||||
| `is_bot` | `bool` | `False` | True when the message author is a bot or webhook (Discord bots). |
|
||||
| `guild_id` | `Optional[str]` | `None` | Discord guild / Slack workspace / Matrix server scope identifier. |
|
||||
| `parent_chat_id` | `Optional[str]` | `None` | Parent channel when `chat_id` refers to a thread. |
|
||||
| `message_id` | `Optional[str]` | `None` | ID of the triggering message. Used for pin/reply/react operations and Discord ID injection. |
|
||||
| `role_authorized` | `bool` | `False` | True when adapter granted access via a platform role (not individual user ID). |
|
||||
|
||||
### Key Methods
|
||||
|
||||
- **`description`** (property: `str`) — Human-readable summary e.g. `"DM with Alice"`,
|
||||
`"group: My Group, thread: 12345"`.
|
||||
- **`to_dict()` / `from_dict()`** — Serialization round-trip for persistence in `sessions.json`.
|
||||
|
||||
---
|
||||
|
||||
## 2. SessionEntry — Active Session Record
|
||||
|
||||
`SessionEntry` is the per-session metadata record stored in memory and persisted to
|
||||
`{sessions_dir}/sessions.json`. Each entry maps a `session_key` to its current `session_id`.
|
||||
|
||||
### Fields
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `session_key` | `str` | *(required)* | Deterministic key identifying the conversation lane (see §4). |
|
||||
| `session_id` | `str` | *(required)* | Unique identifier for this specific conversation incarnation. Format: `YYYYMMDD_HHMMSS_<8hex>`. |
|
||||
| `created_at` | `datetime` | *(required)* | When this session incarnation was created. |
|
||||
| `updated_at` | `datetime` | *(required)* | Last activity timestamp. Used for idle timeout and expiry checks. |
|
||||
| `origin` | `Optional[SessionSource]` | `None` | The source that created this session, used for delivery routing. |
|
||||
| `display_name` | `Optional[str]` | `None` | Chat display name (sourced from `SessionSource.chat_name`). |
|
||||
| `platform` | `Optional[Platform]` | `None` | Platform enum, persisted for expiry policy lookup across restarts. |
|
||||
| `chat_type` | `str` | `"dm"` | Chat type, also persisted for policy lookup. |
|
||||
| `input_tokens` | `int` | `0` | Cumulative LLM input (prompt) tokens consumed. |
|
||||
| `output_tokens` | `int` | `0` | Cumulative LLM output (completion) tokens consumed. |
|
||||
| `cache_read_tokens` | `int` | `0` | Cumulative prompt cache read tokens. |
|
||||
| `cache_write_tokens` | `int` | `0` | Cumulative prompt cache write tokens. |
|
||||
| `total_tokens` | `int` | `0` | Total token count across all turns. |
|
||||
| `estimated_cost_usd` | `float` | `0.0` | Estimated cumulative USD cost. |
|
||||
| `cost_status` | `str` | `"unknown"` | Cost tracking status label. |
|
||||
| `last_prompt_tokens` | `int` | `0` | Last API-reported prompt token count. Used for accurate compression pre-check. |
|
||||
|
||||
### Boolean Flags (State Machine)
|
||||
|
||||
SessionEntry has several boolean flags that form a simple state machine governing session
|
||||
behavior on the next access.
|
||||
|
||||
| Flag | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `was_auto_reset` | `bool` | `False` | Set when a session was auto-reset due to policy expiry (idle/daily). Consumed once to inject a context notice. |
|
||||
| `auto_reset_reason` | `Optional[str]` | `None` | `"idle"` or `"daily"` — why the previous session was auto-reset. |
|
||||
| `reset_had_activity` | `bool` | `False` | Whether the expired session had any messages (`total_tokens > 0`). |
|
||||
| `is_fresh_reset` | `bool` | `False` | Set by explicit `/new` or `/reset`. Triggers topic/channel skill re-injection on first message. Distinguished from `was_auto_reset` to avoid misleading "session expired" notices. |
|
||||
| `expiry_finalized` | `bool` | `False` | Set by background expiry watcher after invoking `on_session_finalize` hooks, cleaning tool resources, and evicting the cached agent. Prevents redundant finalization across restarts. |
|
||||
| `suspended` | `bool` | `False` | Hard force-wipe signal. Set by `/stop` or stuck-loop escalation (3+ consecutive restart failures). On next `get_or_create_session()`, forces a new `session_id` regardless of `resume_pending`. |
|
||||
| `resume_pending` | `bool` | `False` | Soft recovery marker. Set by `suspend_recently_active()` (crash recovery) or drain timeout. On next access, preserves the existing `session_id` — the user continues on the same transcript. Cleared after the next successful turn completes. |
|
||||
| `resume_reason` | `Optional[str]` | `None` | Why resume was marked: `"restart_timeout"`, `"shutdown_timeout"`, `"restart_interrupted"`. |
|
||||
| `last_resume_marked_at` | `Optional[datetime]` | `None` | Timestamp of the last resume-pending marking. |
|
||||
|
||||
### State Transition Logic (get_or_create_session)
|
||||
|
||||
```
|
||||
┌──────────┐
|
||||
│ Incoming │
|
||||
│ Message │
|
||||
└────┬─────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ session_key exists │──── No ──► Create fresh SessionEntry
|
||||
│ AND !force_new │
|
||||
└──────────┬───────────┘
|
||||
│ Yes
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ entry.suspended? │──── Yes ──► Auto-reset: new session_id
|
||||
└──────────┬───────────┘ (reason="suspended")
|
||||
│ No
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ entry.resume_pending?│──── Yes ──► Return existing entry
|
||||
└──────────┬───────────┘ (preserve session_id)
|
||||
│ No Clear flag on next successful turn
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Policy says reset? │──── Yes ──► Auto-reset: new session_id
|
||||
└──────────┬───────────┘ (reason="idle"/"daily")
|
||||
│ No
|
||||
▼
|
||||
┌──────────────────────┐
|
||||
│ Return existing │
|
||||
│ entry, bump │
|
||||
│ updated_at │
|
||||
└──────────────────────┘
|
||||
```
|
||||
|
||||
**Priority order in `get_or_create_session()`:**
|
||||
1. `suspended=True` → always force-reset (hard wipe)
|
||||
2. `resume_pending=True` → preserve session_id (soft recovery)
|
||||
3. Policy expiry (idle/daily) → auto-reset
|
||||
4. No trigger → return existing entry (bump `updated_at`)
|
||||
|
||||
---
|
||||
|
||||
## 3. SessionStore — Storage and Operations
|
||||
|
||||
`SessionStore` is the main storage layer. It maintains an in-memory dict (`_entries`) persisted
|
||||
to `sessions.json`, with SQLite (`SessionDB`) as the canonical store for session metadata and
|
||||
message transcripts.
|
||||
|
||||
### Constructor
|
||||
|
||||
```python
|
||||
SessionStore(sessions_dir: Path, config: GatewayConfig, has_active_processes_fn=None)
|
||||
```
|
||||
|
||||
- `sessions_dir` — Directory where `sessions.json` lives.
|
||||
- `config` — `GatewayConfig` instance for reset policy lookups.
|
||||
- `has_active_processes_fn` — Optional callback keyed by `session_key` to check for running
|
||||
background processes. Sessions with active processes are never expired or pruned.
|
||||
|
||||
### Operations (Methods)
|
||||
|
||||
| Method | Description |
|
||||
|---|---|
|
||||
| `get_or_create_session(source, force_new=False)` | Core entry point. Returns existing or creates new `SessionEntry`. Evaluates `suspended`, `resume_pending`, and reset policy. Creates/ends SQLite records. |
|
||||
| `update_session(session_key, last_prompt_tokens=None)` | Lightweight metadata update after an interaction. Bumps `updated_at`, optionally records `last_prompt_tokens`. |
|
||||
| `reset_session(session_key, display_name=None)` | Explicit reset (from `/new` or `/reset`). Creates new `session_id`, sets `is_fresh_reset=True`. Ends old SQLite session, creates new one. |
|
||||
| `switch_session(session_key, target_session_id)` | Switch to a different existing session ID (from `/resume`). Ends current SQLite session, reopens target. |
|
||||
| `suspend_session(session_key)` | Mark session as `suspended=True` (from `/stop`). Forces auto-reset on next access. |
|
||||
| `mark_resume_pending(session_key, reason)` | Mark session as `resume_pending=True` (from drain timeout). Preserves session_id on next access. Will NOT override `suspended=True`. |
|
||||
| `clear_resume_pending(session_key)` | Clear `resume_pending` after a successful resumed turn. Called from gateway after `run_conversation()` returns. |
|
||||
| `suspend_recently_active(max_age_seconds=120)` | Crash recovery: mark recently-active sessions as `resume_pending=True`. Skips already-pending and already-suspended entries. Called on startup after unclean shutdown. |
|
||||
| `prune_old_entries(max_age_days)` | Drop entries older than `max_age_days` (based on `updated_at`). Skips `suspended` entries and sessions with active processes. |
|
||||
| `list_sessions(active_minutes=None)` | Return all sessions, optionally filtered by recent activity. Sorted by `updated_at` descending. |
|
||||
| `lookup_by_session_id(session_id)` | Find the active `SessionEntry` for a persisted session ID. |
|
||||
| `has_any_sessions()` | Check if any sessions have ever been created (uses SQLite for history, not just in-memory dict). |
|
||||
| `append_to_transcript(session_id, message, skip_db=False)` | Append a message to SQLite transcript. `skip_db=True` prevents duplicate writes when the agent already persisted. |
|
||||
| `rewrite_transcript(session_id, messages)` | Full replacement of session transcript (used by `/retry`, `/undo`, `/compress`). |
|
||||
| `load_transcript(session_id)` | Load all messages from a session's SQLite transcript. |
|
||||
| `rewind_session(session_id, n=1)` | Back up `n` user turns via soft-delete (keeps audit trail). Returns `{rewound_count, turns_undone, target_text}`. |
|
||||
|
||||
### Internal Helpers
|
||||
|
||||
- `_ensure_loaded()` / `_ensure_loaded_locked()` — Load `sessions.json` into `_entries` dict.
|
||||
- `_save()` — Atomic write to `sessions.json` via temp file + `atomic_replace`.
|
||||
- `_generate_session_key(source)` — Delegates to `build_session_key()` with config params.
|
||||
- `_is_session_expired(entry)` — Policy check from entry alone (no source needed). Used by
|
||||
background expiry watcher.
|
||||
- `_should_reset(entry, source)` — Policy check returning `"idle"`, `"daily"`, or `None`.
|
||||
|
||||
### Storage Layout
|
||||
|
||||
```
|
||||
{sessions_dir}/
|
||||
sessions.json # In-memory _entries dict, persisted as JSON
|
||||
Maps session_key → SessionEntry (metadata only)
|
||||
{session_id}.jsonl # (Legacy, removed in spec 002)
|
||||
```
|
||||
|
||||
The canonical transcript store is SQLite via `SessionDB` (from `hermes_state`). The
|
||||
`sessions.json` file persists the `session_key → session_id` mapping and entry metadata
|
||||
(flags, timestamps, token counts). If SQLite is unavailable, the store falls back to
|
||||
JSONL, but this is a degradation path.
|
||||
|
||||
---
|
||||
|
||||
## 4. SessionKey Generation Rules
|
||||
|
||||
Session keys are deterministic strings that identify a conversation lane. They are generated
|
||||
by `build_session_key(source, group_sessions_per_user, thread_sessions_per_user)`.
|
||||
|
||||
### Key Format
|
||||
|
||||
```
|
||||
agent:main:{platform}:{chat_type}[:{chat_id}][:{thread_id}][:{participant_id}]
|
||||
```
|
||||
|
||||
### DM Rules
|
||||
|
||||
| Scenario | Key |
|
||||
|---|---|
|
||||
| DM with chat_id | `agent:main:telegram:dm:12345` |
|
||||
| DM with chat_id + thread | `agent:main:telegram:dm:12345:thread_678` |
|
||||
| DM without chat_id, with participant_id | `agent:main:signal:dm:user_abc` |
|
||||
| DM without chat_id or participant_id | `agent:main:telegram:dm` |
|
||||
| WhatsApp DM (canonicalized) | `agent:main:whatsapp:dm:{canonical_number}` |
|
||||
|
||||
- DMs always include `chat_id` when present, isolating each private conversation.
|
||||
- `thread_id` further differentiates threaded DMs within the same DM chat.
|
||||
- Without `chat_id`, falls back to `user_id_alt` or `user_id` as participant_id.
|
||||
- Without any identifier, all DMs on that platform collapse to one shared session.
|
||||
|
||||
### Group/Channel Rules
|
||||
|
||||
| Scenario | Key |
|
||||
|---|---|
|
||||
| Group chat | `agent:main:telegram:group:-10012345` |
|
||||
| Group chat, per-user isolation | `agent:main:telegram:group:-10012345:user_abc` |
|
||||
| Thread in group, shared | `agent:main:discord:group:12345:thread_678` |
|
||||
| Thread in group, per-user | `agent:main:discord:group:12345:thread_678:user_abc` |
|
||||
| Channel | `agent:main:slack:channel:C12345` |
|
||||
| WhatsApp group (canonicalized) | `agent:main:whatsapp:group:{canonical_id}:{participant}` |
|
||||
|
||||
- `chat_id` identifies the parent group/channel.
|
||||
- `thread_id` differentiates threads within that parent.
|
||||
- **Per-user isolation** (append `participant_id`) is controlled by:
|
||||
- `group_sessions_per_user` (default: `True`) — group/channel sessions are isolated.
|
||||
- `thread_sessions_per_user` (default: `False`) — threads are **shared** by default
|
||||
(Telegram forum topics, Discord threads, Slack threads all share one session per thread).
|
||||
- `participant_id` = `user_id_alt` or `user_id` (in that priority).
|
||||
- WhatsApp identifiers are canonicalized to handle JID/LID alias flips.
|
||||
|
||||
### Special Case: WhatApp
|
||||
|
||||
WhatsApp phone numbers go through `canonical_whatsapp_identifier()` which strips the
|
||||
`@s.whatsapp.net` suffix and normalizes to E.164 format. This prevents session fragmentation
|
||||
when the bridge returns different alias forms of the same phone number.
|
||||
|
||||
---
|
||||
|
||||
## 5. Multi-User Isolation Strategy
|
||||
|
||||
Multi-user isolation determines whether multiple users in the same chat share a conversation
|
||||
or each get their own private session.
|
||||
|
||||
### Decision Logic (`is_shared_multi_user_session`)
|
||||
|
||||
```python
|
||||
def is_shared_multi_user_session(source, *, group_sessions_per_user, thread_sessions_per_user):
|
||||
if source.chat_type == "dm":
|
||||
return False # DMs are always private
|
||||
if source.thread_id:
|
||||
return not thread_sessions_per_user # Threads: shared unless per-user
|
||||
return not group_sessions_per_user # Groups: isolated unless shared
|
||||
```
|
||||
|
||||
### Summary
|
||||
|
||||
| Chat Type | Default | Config Control |
|
||||
|---|---|---|
|
||||
| DM | Private (never shared) | N/A |
|
||||
| Group/Channel | Per-user isolation | `group_sessions_per_user` (default: True) |
|
||||
| Thread (forum, discord) | Shared (all participants see same context) | `thread_sessions_per_user` (default: False) |
|
||||
|
||||
### Impact on System Prompt
|
||||
|
||||
When `shared_multi_user_session=True`, the system prompt omits a fixed user name and instead
|
||||
states: *"Multi-user {thread|session} — messages are prefixed with [sender name]. Multiple
|
||||
users may participate."* Individual sender names are prefixed on each user message by the
|
||||
gateway at runtime, preserving prompt caching (the system prompt doesn't change per-turn).
|
||||
|
||||
---
|
||||
|
||||
## 6. Reset Policy
|
||||
|
||||
Reset policies control when a session automatically loses context (gets a new `session_id`).
|
||||
|
||||
### Policy Modes (`SessionResetPolicy`)
|
||||
|
||||
| Mode | Behavior | Default Config |
|
||||
|---|---|---|
|
||||
| `"none"` | Never auto-reset. Context managed only by compression. | — |
|
||||
| `"idle"` | Reset after N minutes of inactivity from `updated_at`. | `idle_minutes: 1440` (24h) |
|
||||
| `"daily"` | Reset at a specific hour each day (local time). | `at_hour: 4` (4 AM) |
|
||||
| `"both"` | Whichever triggers first — daily boundary OR idle timeout. | **(default)** |
|
||||
|
||||
### Policy Evaluation
|
||||
|
||||
```python
|
||||
# Idle check
|
||||
idle_deadline = entry.updated_at + timedelta(minutes=policy.idle_minutes)
|
||||
if now > idle_deadline: return "idle"
|
||||
|
||||
# Daily check
|
||||
today_reset = now.replace(hour=policy.at_hour, minute=0, second=0, microsecond=0)
|
||||
if now.hour < policy.at_hour:
|
||||
today_reset -= timedelta(days=1) # Reset hasn't happened yet today
|
||||
if entry.updated_at < today_reset: return "daily"
|
||||
```
|
||||
|
||||
### Per-Platform/Per-Type Policies
|
||||
|
||||
Reset policies are configurable per platform and session type via `config.get_reset_policy()`.
|
||||
This allows different platforms to have different expiry rules (e.g., Telegram DMs reset
|
||||
after 24h idle, but Slack groups persist indefinitely).
|
||||
|
||||
### Exclusions
|
||||
|
||||
Sessions with active background processes are **never** expired or reset. The
|
||||
`has_active_processes_fn` callback checks for running processes when evaluating policies.
|
||||
|
||||
### Reset Effects
|
||||
|
||||
When a reset triggers:
|
||||
|
||||
1. Old session is ended in SQLite (with reason `"session_reset"`).
|
||||
2. New `session_id` is generated (`YYYYMMDD_HHMMSS_<8hex>`).
|
||||
3. New `SessionEntry` is created with `was_auto_reset=True` and the reset reason.
|
||||
4. `reset_had_activity` is set if the old session had any turns (`total_tokens > 0`).
|
||||
5. The old AIAgent cache entry is evicted on the next expiry watcher pass.
|
||||
6. On the first message after reset, a context notice is injected: "Session expired due to inactivity / daily reset."
|
||||
|
||||
---
|
||||
|
||||
## 7. Restart Recovery Flow
|
||||
|
||||
The restart recovery system ensures that in-flight sessions are preserved across gateway
|
||||
restarts, crashes, and drain timeouts. It is the solution to issue #7536.
|
||||
|
||||
### Startup Recovery Sequence
|
||||
|
||||
```
|
||||
Gateway starts
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────┐
|
||||
│ Check for .clean_shutdown │── Exists? ──► Skip suspension (clean exit)
|
||||
│ marker │
|
||||
└───────────────────────────────┘
|
||||
│ Missing
|
||||
▼
|
||||
┌───────────────────────────────┐
|
||||
│ session_store │── Marks sessions updated within
|
||||
│ .suspend_recently_active() │ last 120 seconds as resume_pending
|
||||
└───────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────┐
|
||||
│ _suspend_stuck_loop_sessions()│── Suspends sessions that have been
|
||||
│ │ active across 3+ restarts
|
||||
└───────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────┐
|
||||
│ Queue inbound messages while │
|
||||
│ startup restore runs │
|
||||
│ (_startup_restore_in_progress)│
|
||||
└───────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────┐
|
||||
│ For each adapter, find │
|
||||
│ resume_pending sessions → │
|
||||
│ synthesize MessageEvent and │
|
||||
│ run _handle_message to let │
|
||||
│ the agent auto-continue │
|
||||
└───────────────────────────────┘
|
||||
```
|
||||
|
||||
### suspend_recently_active(max_age_seconds=120)
|
||||
|
||||
Called on gateway startup when no `.clean_shutdown` marker exists (indicating a crash or
|
||||
unexpected exit). For each session updated within the last 120 seconds:
|
||||
|
||||
- Sets `resume_pending=True`, `resume_reason="restart_interrupted"`,
|
||||
`last_resume_marked_at=now`.
|
||||
- Skips entries already `resume_pending=True` (no double-mark).
|
||||
- Skips entries explicitly `suspended=True` (hard wipe should stay).
|
||||
|
||||
### Stuck-Loop Detection (`_suspend_stuck_loop_sessions`)
|
||||
|
||||
Counts consecutive restarts via a JSON file (`{HERMES_HOME}/restart_counts.json`). If a
|
||||
session has been active across 3+ consecutive restarts, it's auto-suspended so the user
|
||||
gets a clean slate.
|
||||
|
||||
### Drain-Timeout Marking
|
||||
|
||||
On graceful shutdown/restart, the drain system calls `mark_resume_pending()` for any
|
||||
session that was mid-turn when the drain timeout fired. Reasons:
|
||||
|
||||
- `"restart_timeout"` — killed during restart drain
|
||||
- `"shutdown_timeout"` — killed during shutdown drain
|
||||
- `"restart_interrupted"` — crash recovery (from `suspend_recently_active`)
|
||||
|
||||
All three reasons are in `_AUTO_RESUME_REASONS` and eligible for startup auto-resume.
|
||||
|
||||
### Auto-Resume on Next Access
|
||||
|
||||
When `get_or_create_session()` encounters `resume_pending=True`:
|
||||
|
||||
1. It returns the existing entry **without** creating a new `session_id`.
|
||||
2. The existing transcript is loaded intact.
|
||||
3. The marking is not cleared here — it survives until the next successful turn
|
||||
completes (`clear_resume_pending()` is called from the gateway after
|
||||
`run_conversation()` returns a real response).
|
||||
4. If the resumed turn is interrupted again, the `resume_pending` flag remains set,
|
||||
and the next restart will retry. The stuck-loop counter handles terminal escalation
|
||||
(3 retries → suspended).
|
||||
|
||||
### Clean Shutdown Marker (`.clean_shutdown`)
|
||||
|
||||
Written at the end of a graceful shutdown. On next startup:
|
||||
|
||||
- If present: skip `suspend_recently_active()` entirely. Active agents were already
|
||||
drained, so no sessions are stuck.
|
||||
- Then delete the marker.
|
||||
|
||||
This prevents unwanted auto-resets after `hermes update`, `hermes gateway restart`,
|
||||
or `/restart`.
|
||||
|
||||
---
|
||||
|
||||
## 8. Message Queuing Flow
|
||||
|
||||
The message queuing system handles two scenarios:
|
||||
|
||||
1. **Interrupt follow-ups** — When a user sends multiple messages while the agent is
|
||||
processing, subsequent messages are queued as single-slot pending messages.
|
||||
2. **`/queue` FIFO** — Explicit `/queue` commands that must each produce their own full
|
||||
agent turn, in order, without merging.
|
||||
|
||||
### Data Structures
|
||||
|
||||
```
|
||||
adapter._pending_messages: Dict[session_key, MessageEvent]
|
||||
└── Single "next-up" slot per session. Overwritten on repeat sends
|
||||
(burst collapse). Shared with photo-burst follow-ups.
|
||||
|
||||
self._queued_events: Dict[session_key, List[MessageEvent]]
|
||||
└── Overflow buffer. Each /queue invocation appends here when the
|
||||
slot is occupied. Promoted one-at-a-time after each drain.
|
||||
```
|
||||
|
||||
### Enqueue (`_enqueue_fifo`)
|
||||
|
||||
```
|
||||
_enqueue_fifo(session_key, event, adapter)
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────────────┐
|
||||
│ Is slot free? │
|
||||
│ (session_key NOT in _pending_messages)│── Yes ──► Place event in slot
|
||||
└───────────────────────────────────────┘
|
||||
│ No
|
||||
▼
|
||||
Append to _queued_events[session_key] (overflow tail)
|
||||
```
|
||||
|
||||
### Dequeue / Promotion (`_promote_queued_event`)
|
||||
|
||||
Called at the drain site after the slot was consumed. If there's an overflow item:
|
||||
|
||||
- When `pending_event is None` (slot was empty), return overflow head as the new event.
|
||||
- When `pending_event` exists, stage overflow head in the slot for the next recursion.
|
||||
- If no adapter available, push back to `_queued_events` (don't silently drop).
|
||||
|
||||
### Queue Depth
|
||||
|
||||
`_queue_depth(session_key, adapter)` returns `len(overflow) + (1 if slot occupied else 0)`.
|
||||
|
||||
### Clearing
|
||||
|
||||
Queued events for a session are cleared on `/new` and `/reset` (via `_handle_reset_command`).
|
||||
|
||||
### FIFO Invariant
|
||||
|
||||
Each `/queue` invocation produces exactly one full agent turn, in FIFO order, with no
|
||||
merging. The single-slot `_pending_messages` + overflow `_queued_events` design ensures
|
||||
that repeated sends during an active turn don't cause out-of-order processing.
|
||||
|
||||
---
|
||||
|
||||
## 9. Session Context Injection
|
||||
|
||||
`SessionContext` is built from a `SessionSource` and `GatewayConfig` and injected into the
|
||||
agent's system prompt. It tells the agent:
|
||||
|
||||
- Where the current message came from
|
||||
- What platforms are connected
|
||||
- Where it can deliver scheduled task outputs
|
||||
- Whether this is a shared multi-user session
|
||||
|
||||
### Construction (`build_session_context`)
|
||||
|
||||
```python
|
||||
def build_session_context(source, config, session_entry=None) -> SessionContext
|
||||
```
|
||||
|
||||
1. Collects connected platforms from config.
|
||||
2. Collects home channels for each platform.
|
||||
3. Determines `shared_multi_user_session` via `is_shared_multi_user_session()`.
|
||||
4. Attaches session metadata (key, id, timestamps) if `session_entry` is provided.
|
||||
|
||||
### PII Redaction (`build_session_context_prompt`)
|
||||
|
||||
The dynamic system prompt section (`## Current Session Context`) can optionally redact
|
||||
personally identifiable information before sending to the LLM:
|
||||
|
||||
- User IDs → `user_<12hex>` (SHA-256 prefix)
|
||||
- Chat IDs → `<platform>:<12hex>` or just `<12hex>`
|
||||
- Platforms excluded from redaction: Discord (needs raw IDs for `@mentions`),
|
||||
and any plugin-registered platform not marked `pii_safe`.
|
||||
|
||||
Redaction applies only to the system prompt text. Routing, session keys, and adapter
|
||||
operations always use the original values.
|
||||
|
||||
---
|
||||
|
||||
## 10. Background Expiry Watcher
|
||||
|
||||
The `_session_expiry_watcher` task runs in the gateway event loop every 300 seconds (5 min).
|
||||
|
||||
### Responsibilities
|
||||
|
||||
1. **Finalize expired sessions** — For each entry where `_is_session_expired()` returns
|
||||
True and `expiry_finalized` is False:
|
||||
- Invoke `on_session_finalize` plugin hooks (cleanup, notifications).
|
||||
- Clean up cached AIAgent resources (close tool resources, shut down memory provider).
|
||||
- Evict the cached agent entry.
|
||||
- Clear per-session overrides (`_session_model_overrides`, reasoning overrides, etc.).
|
||||
- Mark `expiry_finalized=True` and persist.
|
||||
|
||||
2. **Sweep idle cached agents** — Calls `_sweep_idle_cached_agents()` to evict agents that
|
||||
have been idle beyond `_AGENT_CACHE_IDLE_TTL_SECS` (3600s / 1h), regardless of session
|
||||
reset policy. This prevents unbounded memory growth in gateways with long-lived sessions.
|
||||
|
||||
3. **Prune stale entries** — Calls `session_store.prune_old_entries()` hourly based on
|
||||
`config.session_store_max_age_days`. Prevents `sessions.json` from growing unbounded.
|
||||
|
||||
### Failure Handling
|
||||
|
||||
- Per-session retry count: each failed finalize is retried up to 3 consecutive times.
|
||||
- After 3 failures, the entry is force-marked `expiry_finalized=True` to prevent infinite
|
||||
retry loops.
|
||||
|
||||
---
|
||||
|
||||
## 11. Agent Cache
|
||||
|
||||
The gateway maintains an LRU cache of `AIAgent` instances keyed by `session_key` to
|
||||
preserve prompt caching across turns.
|
||||
|
||||
### Cache Properties
|
||||
|
||||
- **Max size:** 128 entries (`_AGENT_CACHE_MAX_SIZE`).
|
||||
- **Eviction policy:** Least-recently-used (LRU via `OrderedDict`).
|
||||
- **Idle TTL:** 3600s (1h) — enforced by `_session_expiry_watcher`.
|
||||
- **Lock:** `_agent_cache_lock` (threading) for thread safety.
|
||||
|
||||
### Cache Lifecycle
|
||||
|
||||
```
|
||||
Message arrives
|
||||
│
|
||||
▼
|
||||
get_or_create_session() → session_key obtained
|
||||
│
|
||||
▼
|
||||
Lookup _agent_cache[session_key]
|
||||
│
|
||||
├── Hit → move_to_end(), reuse AIAgent (preserves prompt cache)
|
||||
│
|
||||
└── Miss → create new AIAgent, store in cache
|
||||
(if at capacity, popitem(last=False) evicts LRU entry)
|
||||
│
|
||||
▼
|
||||
run_conversation() → agent processes message
|
||||
│
|
||||
▼
|
||||
Session expiry watcher evicts agent when session finalizes
|
||||
```
|
||||
|
||||
### Cleanup Flow
|
||||
|
||||
When a session expires:
|
||||
1. `_cleanup_agent_resources(agent)` — shuts down memory provider, closes tool resources.
|
||||
2. `_evict_cached_agent(key)` — removes from `_agent_cache` so the agent can be GC'd.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Key Configuration
|
||||
|
||||
| Config Key | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `group_sessions_per_user` | `bool` | `true` | Isolate group/channel sessions per user |
|
||||
| `thread_sessions_per_user` | `bool` | `false` | Isolate thread sessions per user |
|
||||
| `session_store_max_age_days` | `int` | `0` | Prune sessions older than N days (0=disabled) |
|
||||
| `agent.gateway_auto_continue_freshness` | `int` | `3600` | Seconds for resume freshness window |
|
||||
| `agent.gateway_timeout` | `int` | `1800` | Agent turn timeout (30 min default) |
|
||||
|
||||
### Reset Policy (per-platform/type, in config.yaml)
|
||||
|
||||
```yaml
|
||||
session_reset:
|
||||
mode: both # none | idle | daily | both
|
||||
at_hour: 4 # daily reset hour (local time)
|
||||
idle_minutes: 1440 # idle timeout (24h)
|
||||
notify: true # notify user on auto-reset
|
||||
```
|
||||
|
||||
Platform-specific overrides can be set under `platforms.<name>.session_reset`.
|
||||
Reference in New Issue
Block a user