# Reflection Reflection periodically consolidates the memory fragments scattered across many conversations into a single, chronologically-organized narrative. It runs offline in the background: it merges the multiple Episodes inside one similarity cluster into one, resolves stale information by keeping the latest state, and soft-archives the originals it replaces — so memory gets more accurate and more compact with use, instead of piling up into noise. ## Prerequisites Reflection's merge step calls an LLM, and re-clustering the merged narrative calls an embedding model. Configure both as you would for the rest of EverOS — an OpenAI-compatible `[llm]` and `[embedding]` block in `/everos.toml`, or the matching `EVEROS_LLM__*` / `EVEROS_EMBEDDING__*` environment variables. If a provider is unavailable, the affected clusters are skipped and logged rather than failing the run. ## Quick start > The examples below assume EverOS is running on the default port 8000. > `` is the EverOS memory root (see [QUICKSTART](../QUICKSTART.md)). Reflection is off by default. Turn it on in `/ome.toml` — **one line**: ```toml [strategies.reflect_episodes] enabled = true ``` Once enabled, it **runs automatically every Monday at 02:00**. This is how Reflection is meant to work — nothing else to configure. To change the run time, add a `cron` line (a schedule expression; optional): ```toml [strategies.reflect_episodes] enabled = true cron = "0 3 * * 0" # optional: change the run time (here: Sundays at 03:00) ``` > **Don't run it too often.** Once a week at most is recommended. Each run > is a lossy LLM merge; repeatedly re-consolidating the same memories can > make the narrative *worse*, not better — which is why the default is > weekly. Config changes hot-reload (no restart needed). From then on, at each scheduled time, Reflection consolidates each user's memory once. **What does it produce, and where do I see it?** Each run **appends one merged narrative** to the relevant user's Episode log, and marks the older fragments it replaces as archived (removed from default search). Markdown is the source of truth — just open the user's Episode log file: ``` /default_app/default_project/users//episodes/episode-.md ``` The new entry carries `parent_type: cluster` (Episodes produced by ordinary conversation are `parent_type: memcell`). It looks roughly like this: ```markdown --- owner_id: u_andrew timestamp: 2026-10-11T02:00:00+00:00 parent_type: cluster # <- marks it as a Reflection merge product parent_id: cl_a1b2c3d4e5f6 --- ## Subject Andrew's pet adoption journey ## Content Andrew initially had no pets. He later adopted a dog named Toby, and then adopted another dog named Buddy. He currently has two dogs. ``` A search on the topic afterwards returns this single complete narrative rather than the scattered old fragments. > To inspect which clusters were consolidated, how many entries were > archived, etc., see [Auditing & troubleshooting](#auditing--troubleshooting) > (advanced; not needed for everyday use). > For debugging without waiting for the schedule, you can trigger a run by > hand — see [Triggering a run](#triggering-a-run). ## How it works Reflection runs *offline*, separate from the live conversation path. The online path keeps extracting Episodes and clustering them; Reflection later consumes those clusters — it never sits between a user and a response: ``` Online (never blocked) Offline (scheduled) ────────────────────── ─────────────────── conversation → Episode → Cluster ───► Reflection consolidates the clusters ``` A scheduled run processes every user across all app/project tenants that have clusters. After each conversation, EverOS extracts an **Episode** (a summary of a conversation segment), and geometric clustering groups semantically similar, time-adjacent Episodes into a **Cluster**. The same topic thus ends up scattered as several point-in-time snapshots within one cluster: ``` Cluster cl_xxx ├── ep_0001 "Andrew has no pets yet" (August) ├── ep_0002 "Andrew adopted Toby" (September) └── ep_0003 "Andrew also adopted Buddy" (October) ``` Reflection consolidates one cluster at a time, in four steps: ``` Select ─→ Merge ─→ Re-extract ─→ Deprecate ``` 1. **Select** — pick clusters worth consolidating: not yet consolidated and holding ≥ 2 members, or already consolidated and since joined by new members. At most 10 clusters per run, largest first. 2. **Merge** — hand the cluster's Episodes to the LLM in chronological order and merge them into one narrative: preserve facts, resolve contradictions by keeping the latest state, restore the timeline, drop duplicates, and end with the current state. A previously consolidated cluster is updated incrementally — only the new fragments are folded into the existing narrative. 3. **Re-extract** — the merged narrative is written to Markdown and triggers re-extraction of atomic facts, keeping derived data consistent with it. 4. **Deprecate** — the replaced original Episodes and their atomic facts get `deprecated_by` pointing at the new narrative; cluster membership is updated; an audit record is written. The result: ``` Cluster cl_xxx └── ep_0042 "Andrew initially had no pets. He later adopted a dog named Toby, then another named Buddy. He currently has two dogs." (originals ep_0001 / ep_0002 / ep_0003 → deprecated_by = ep_0042) ``` The merged narrative is, structurally, just an ordinary Episode (`parent_type="cluster"`) — transparent to retrieval, no search-pipeline changes required. Default search excludes any memory carrying `deprecated_by`, so a query like "how many pets does Andrew have" only hits the one complete narrative. ## Storage layout Memory uses Markdown as the single source of truth; SQLite and LanceDB are derived indexes built automatically by the cascade daemon. | Store | What it holds | Role | |---|---|---| | Markdown | Episode bodies, merged narratives, archive markers | Single source of truth; human-readable and editable | | SQLite | Clusters and members, consolidation audit records | Structured state and queries | | LanceDB | Vectors + BM25 index for Episodes / atomic facts | Search (rebuildable from Markdown) | The **merged narrative** is written to the Episode daily-log Markdown; its frontmatter marks that it came from a cluster: ```yaml --- owner_id: u_andrew timestamp: 2026-10-10T12:00:00+00:00 parent_type: cluster parent_id: cl_a1b2c3d4e5f6 --- ## Subject Andrew's pet adoption journey ## Content Andrew initially had no pets. He later adopted a dog named Toby, and then adopted another dog named Buddy. He currently has two dogs. ``` The **replaced originals** are not deleted. Their file's frontmatter records the archive mapping, and the index layer writes `deprecated_by`: ```yaml --- # added to the original Episode file's frontmatter: deprecated_entries: ep_20260810_0001: ep_20261010_0042 ep_20260910_0002: ep_20261010_0042 --- ``` > Soft-archive, not delete: even if SQLite / LanceDB are corrupted, as long > as the Markdown is intact the indexes can be fully rebuilt — and every > consolidation remains traceable back to its original content. ## Configuration | Setting | Location | Default | Description | |---|---|---|---| | `reflect_episodes.enabled` | `/ome.toml` | `false` | Set to `true` to enable (the only setting needed) | | `reflect_episodes.cron` | `/ome.toml` | `0 2 * * 1` | Run time, as a standard cron expression (`0 2 * * 1` = Mondays at 02:00); **optional**, omit to use the built-in default. Running more than weekly is not recommended | | `clustering.threshold` | `/everos.toml` | `0.65` | Clustering similarity threshold | | `clustering.time_window_days` | `/everos.toml` | `7.0` | Clustering time window (days) | Two files, two scopes: `ome.toml` holds OME-strategy config (Reflection's on/off switch and schedule); `everos.toml` holds general settings (clustering and the like). Both live under the memory root, and you only write the keys you want to override — everything else falls back to the shipped defaults in `config/default.toml`, which you never edit by hand. Changes to `ome.toml` hot-reload (~1–2s); no server restart needed. > Setting `enabled` back to `false` stops the *next* run from starting; a run > already in progress finishes normally. ## API reference Reflection's normal mode of operation is the scheduled automatic run (see [Quick start](#quick-start)). The endpoint below triggers a run **on demand** — for testing, debugging, or when you want to consolidate immediately. It is an auxiliary path, not the normal mode (there is no dedicated CLI command). ### Triggering a run ``` POST /api/v1/ome/trigger Content-Type: application/json ``` | Field | Type | Default | Description | |---|---|---|---| | `name` | string | — | Strategy name; use `reflect_episodes` | | `timeout` | float | 120.0 | Max seconds to wait for the run to finish | | `force` | bool | false | When `true`, runs even if `enabled=false` | **Response**: ```json { "status": "ok", "name": "reflect_episodes" } ``` `status` is `"ok"` (finished) or `"timeout"` (did not finish in time); an unknown strategy name returns 404. **Example — Python**: ```python import httpx async def trigger_reflection() -> str: async with httpx.AsyncClient(base_url="http://localhost:8000") as client: resp = await client.post( "/api/v1/ome/trigger", json={"name": "reflect_episodes", "timeout": 120, "force": True}, ) resp.raise_for_status() return resp.json()["status"] ``` **Example — curl**: ```bash curl -X POST http://localhost:8000/api/v1/ome/trigger \ -H "Content-Type: application/json" \ -d '{"name": "reflect_episodes", "timeout": 120, "force": true}' ``` ## Auditing & troubleshooting > This section is **advanced**. For everyday use you don't need it — just > read the Markdown (see [Quick start](#quick-start)). It's here for > inspecting consolidation details or diagnosing problems. Each run writes one `reflection_report` audit record, useful for reviewing consolidation history: | Field | Description | |---|---| | `cluster_id` | The cluster that was consolidated | | `mode` | `init` (first merge) or `update` (incremental update) | | `source_count` | Number of fragments merged | | `merged_entry_id` | The merged-narrative Episode produced | | `deprecated_fact_count` | Number of atomic facts archived alongside | | `created_at` | Consolidation time | ```bash sqlite3 /.index/sqlite/system.db \ "SELECT cluster_id, mode, source_count, merged_entry_id FROM reflection_report ORDER BY created_at DESC LIMIT 10;" ``` | Symptom | Likely cause | |---|---| | No consolidation records after triggering | No eligible clusters (a cluster needs ≥ 2 members) | | Response `status: "timeout"` | Downstream re-extraction is slow; raise `timeout` and retry | | Old fragments still appear in search | Index syncs asynchronously, usually 1–3s; wait and retry | | 404 returned | Strategy name must be `reflect_episodes` | ## Design notes Why Reflection is shaped the way it is: - **Offline and scheduled.** Merging is a heavy, lossy LLM operation, so it runs off the request path — conversations stay fast — and a weekly cadence lets enough new fragments accumulate to be worth re-merging. - **Soft-archive, never delete.** Originals stay in Markdown, so every consolidation is traceable and the indexes can always be rebuilt from the Markdown source of truth. - **A merged narrative is just an Episode.** Reusing the Episode type means search and every downstream consumer keep working unchanged — Reflection introduces no new retrieval path. ## Limitations - **Merging is lossy** — LLM consolidation may drop individual details. The original fragments are retained in storage and remain traceable, but are not in default search results. - **Clustering is by similarity** — Reflection consolidates the output of similarity clustering; a single cluster is not guaranteed to be strictly one topic. - **No one-click rollback yet** — originals are fully retained, but there is currently no endpoint to undo a specific consolidation. ## End-to-end walkthrough The walkthrough triggers a run by hand to demonstrate the full flow; in a real deployment, once enabled it runs automatically on schedule, so this step isn't needed. ```bash BASE=http://localhost:8000/api/v1 # 1. With Reflection enabled (set enabled = true in /ome.toml), # trigger a run by hand here (for the demo; in production it runs on schedule) curl -s -X POST "$BASE/ome/trigger" \ -H "Content-Type: application/json" \ -d '{"name": "reflect_episodes", "timeout": 120, "force": true}' \ | jq . # → { "status": "ok", "name": "reflect_episodes" } # 2. Review the consolidation audit sqlite3 /.index/sqlite/system.db \ "SELECT mode, source_count, merged_entry_id FROM reflection_report ORDER BY created_at DESC LIMIT 1;" # → init|3|ep_20261010_0042 # 3. Verify via search: the hit is the merged narrative, not the old fragments curl -s -X POST "$BASE/memory/search" \ -H "Content-Type: application/json" \ -d '{"query": "how many pets does Andrew have", "top_k": 5}' \ | jq '.data.episodes[0] | {subject, episode, session_id}' # → session_id is null on a merged narrative (the aggregation-product marker); # episode holds the full narrative text ``` ## See also - [how-memory-works.md](how-memory-works.md) — Episodes and the memory extraction pipeline - [storage_layout.md](storage_layout.md) — Markdown + SQLite + LanceDB stack - [api.md](api.md) — full HTTP API reference