evermind-ai--everos/docs/reflection.md

Reflection

Reflection periodically consolidates the memory fragments scattered across many conversations into a single, chronologically-organized narrative. It runs offline in the background: it merges the multiple Episodes inside one similarity cluster into one, resolves stale information by keeping the latest state, and soft-archives the originals it replaces — so memory gets more accurate and more compact with use, instead of piling up into noise.

Prerequisites

Reflection's merge step calls an LLM, and re-clustering the merged narrative calls an embedding model. Configure both as you would for the rest of EverOS — an OpenAI-compatible [llm] and [embedding] block in <root>/everos.toml, or the matching EVEROS_LLM__* / EVEROS_EMBEDDING__* environment variables. If a provider is unavailable, the affected clusters are skipped and logged rather than failing the run.

Quick start

The examples below assume EverOS is running on the default port 8000. <root> is the EverOS memory root (see QUICKSTART).

Reflection is off by default. Turn it on in <root>/ome.toml — one line:

[strategies.reflect_episodes]
enabled = true

Once enabled, it runs automatically every Monday at 02:00. This is how Reflection is meant to work — nothing else to configure. To change the run time, add a cron line (a schedule expression; optional):

[strategies.reflect_episodes]
enabled = true
cron = "0 3 * * 0"        # optional: change the run time (here: Sundays at 03:00)

Don't run it too often. Once a week at most is recommended. Each run is a lossy LLM merge; repeatedly re-consolidating the same memories can make the narrative worse, not better — which is why the default is weekly.

Config changes hot-reload (no restart needed). From then on, at each scheduled time, Reflection consolidates each user's memory once.

What does it produce, and where do I see it? Each run appends one merged narrative to the relevant user's Episode log, and marks the older fragments it replaces as archived (removed from default search). Markdown is the source of truth — just open the user's Episode log file:

<root>/default_app/default_project/users/<user_id>/episodes/episode-<date>.md

The new entry carries parent_type: cluster (Episodes produced by ordinary conversation are parent_type: memcell). It looks roughly like this:

---
owner_id: u_andrew
timestamp: 2026-10-11T02:00:00+00:00
parent_type: cluster          # <- marks it as a Reflection merge product
parent_id: cl_a1b2c3d4e5f6
---
## Subject
Andrew's pet adoption journey

## Content
Andrew initially had no pets. He later adopted a dog named Toby, and then
adopted another dog named Buddy. He currently has two dogs.

A search on the topic afterwards returns this single complete narrative rather than the scattered old fragments.

To inspect which clusters were consolidated, how many entries were archived, etc., see Auditing & troubleshooting (advanced; not needed for everyday use). For debugging without waiting for the schedule, you can trigger a run by hand — see Triggering a run.

How it works

Reflection runs offline, separate from the live conversation path. The online path keeps extracting Episodes and clustering them; Reflection later consumes those clusters — it never sits between a user and a response:

Online (never blocked)                Offline (scheduled)
──────────────────────                ───────────────────
conversation → Episode → Cluster ───► Reflection consolidates the clusters

A scheduled run processes every user across all app/project tenants that have clusters.

After each conversation, EverOS extracts an Episode (a summary of a conversation segment), and geometric clustering groups semantically similar, time-adjacent Episodes into a Cluster. The same topic thus ends up scattered as several point-in-time snapshots within one cluster:

Cluster cl_xxx
├── ep_0001  "Andrew has no pets yet"       (August)
├── ep_0002  "Andrew adopted Toby"          (September)
└── ep_0003  "Andrew also adopted Buddy"    (October)

Reflection consolidates one cluster at a time, in four steps:

Select ─→ Merge ─→ Re-extract ─→ Deprecate

1. Select — pick clusters worth consolidating: not yet consolidated and holding ≥ 2 members, or already consolidated and since joined by new members. At most 10 clusters per run, largest first.
2. Merge — hand the cluster's Episodes to the LLM in chronological order and merge them into one narrative: preserve facts, resolve contradictions by keeping the latest state, restore the timeline, drop duplicates, and end with the current state. A previously consolidated cluster is updated incrementally — only the new fragments are folded into the existing narrative.
3. Re-extract — the merged narrative is written to Markdown and triggers re-extraction of atomic facts, keeping derived data consistent with it.
4. Deprecate — the replaced original Episodes and their atomic facts get deprecated_by pointing at the new narrative; cluster membership is updated; an audit record is written.

The result:

Cluster cl_xxx
└── ep_0042  "Andrew initially had no pets. He later adopted a dog named
              Toby, then another named Buddy. He currently has two dogs."
              (originals ep_0001 / ep_0002 / ep_0003 → deprecated_by = ep_0042)

The merged narrative is, structurally, just an ordinary Episode (parent_type="cluster") — transparent to retrieval, no search-pipeline changes required. Default search excludes any memory carrying deprecated_by, so a query like "how many pets does Andrew have" only hits the one complete narrative.

Storage layout

Memory uses Markdown as the single source of truth; SQLite and LanceDB are derived indexes built automatically by the cascade daemon.

Store	What it holds	Role
Markdown	Episode bodies, merged narratives, archive markers	Single source of truth; human-readable and editable
SQLite	Clusters and members, consolidation audit records	Structured state and queries
LanceDB	Vectors + BM25 index for Episodes / atomic facts	Search (rebuildable from Markdown)

The merged narrative is written to the Episode daily-log Markdown; its frontmatter marks that it came from a cluster:

---
owner_id: u_andrew
timestamp: 2026-10-10T12:00:00+00:00
parent_type: cluster
parent_id: cl_a1b2c3d4e5f6
---
## Subject
Andrew's pet adoption journey

## Content
Andrew initially had no pets. He later adopted a dog named Toby, and then
adopted another dog named Buddy. He currently has two dogs.

The replaced originals are not deleted. Their file's frontmatter records the archive mapping, and the index layer writes deprecated_by:

---
# added to the original Episode file's frontmatter:
deprecated_entries:
  ep_20260810_0001: ep_20261010_0042
  ep_20260910_0002: ep_20261010_0042
---

Soft-archive, not delete: even if SQLite / LanceDB are corrupted, as long as the Markdown is intact the indexes can be fully rebuilt — and every consolidation remains traceable back to its original content.

Configuration

Setting	Location	Default	Description
reflect_episodes.enabled	<root>/ome.toml	false	Set to true to enable (the only setting needed)
reflect_episodes.cron	<root>/ome.toml	0 2 * * 1	Run time, as a standard cron expression (0 2 * * 1 = Mondays at 02:00); optional, omit to use the built-in default. Running more than weekly is not recommended
clustering.threshold	<root>/everos.toml	0.65	Clustering similarity threshold
clustering.time_window_days	<root>/everos.toml	7.0	Clustering time window (days)

Two files, two scopes: ome.toml holds OME-strategy config (Reflection's on/off switch and schedule); everos.toml holds general settings (clustering and the like). Both live under the memory root, and you only write the keys you want to override — everything else falls back to the shipped defaults in config/default.toml, which you never edit by hand.

Changes to ome.toml hot-reload (~1–2s); no server restart needed.

Setting enabled back to false stops the next run from starting; a run already in progress finishes normally.

API reference

Reflection's normal mode of operation is the scheduled automatic run (see Quick start). The endpoint below triggers a run on demand — for testing, debugging, or when you want to consolidate immediately. It is an auxiliary path, not the normal mode (there is no dedicated CLI command).

Triggering a run

POST /api/v1/ome/trigger
Content-Type: application/json

Field	Type	Default	Description
name	string	—	Strategy name; use reflect_episodes
timeout	float	120.0	Max seconds to wait for the run to finish
force	bool	false	When true, runs even if enabled=false

Response:

{ "status": "ok", "name": "reflect_episodes" }

status is "ok" (finished) or "timeout" (did not finish in time); an unknown strategy name returns 404.

Example — Python:

import httpx


async def trigger_reflection() -> str:
    async with httpx.AsyncClient(base_url="http://localhost:8000") as client:
        resp = await client.post(
            "/api/v1/ome/trigger",
            json={"name": "reflect_episodes", "timeout": 120, "force": True},
        )
        resp.raise_for_status()
        return resp.json()["status"]

Example — curl:

curl -X POST http://localhost:8000/api/v1/ome/trigger \
  -H "Content-Type: application/json" \
  -d '{"name": "reflect_episodes", "timeout": 120, "force": true}'

Auditing & troubleshooting

This section is advanced. For everyday use you don't need it — just read the Markdown (see Quick start). It's here for inspecting consolidation details or diagnosing problems.

Each run writes one reflection_report audit record, useful for reviewing consolidation history:

Field	Description
cluster_id	The cluster that was consolidated
mode	init (first merge) or update (incremental update)
source_count	Number of fragments merged
merged_entry_id	The merged-narrative Episode produced
deprecated_fact_count	Number of atomic facts archived alongside
created_at	Consolidation time

sqlite3 <root>/.index/sqlite/system.db \
  "SELECT cluster_id, mode, source_count, merged_entry_id
   FROM reflection_report ORDER BY created_at DESC LIMIT 10;"

Symptom	Likely cause
No consolidation records after triggering	No eligible clusters (a cluster needs ≥ 2 members)
Response status: "timeout"	Downstream re-extraction is slow; raise timeout and retry
Old fragments still appear in search	Index syncs asynchronously, usually 1–3s; wait and retry
404 returned	Strategy name must be reflect_episodes

Design notes

Why Reflection is shaped the way it is:

• Offline and scheduled. Merging is a heavy, lossy LLM operation, so it runs off the request path — conversations stay fast — and a weekly cadence lets enough new fragments accumulate to be worth re-merging.
• Soft-archive, never delete. Originals stay in Markdown, so every consolidation is traceable and the indexes can always be rebuilt from the Markdown source of truth.
• A merged narrative is just an Episode. Reusing the Episode type means search and every downstream consumer keep working unchanged — Reflection introduces no new retrieval path.

Limitations

• Merging is lossy — LLM consolidation may drop individual details. The original fragments are retained in storage and remain traceable, but are not in default search results.
• Clustering is by similarity — Reflection consolidates the output of similarity clustering; a single cluster is not guaranteed to be strictly one topic.
• No one-click rollback yet — originals are fully retained, but there is currently no endpoint to undo a specific consolidation.

End-to-end walkthrough

The walkthrough triggers a run by hand to demonstrate the full flow; in a real deployment, once enabled it runs automatically on schedule, so this step isn't needed.

BASE=http://localhost:8000/api/v1

# 1. With Reflection enabled (set enabled = true in <root>/ome.toml),
#    trigger a run by hand here (for the demo; in production it runs on schedule)
curl -s -X POST "$BASE/ome/trigger" \
  -H "Content-Type: application/json" \
  -d '{"name": "reflect_episodes", "timeout": 120, "force": true}' \
  | jq .
# → { "status": "ok", "name": "reflect_episodes" }

# 2. Review the consolidation audit
sqlite3 <root>/.index/sqlite/system.db \
  "SELECT mode, source_count, merged_entry_id
   FROM reflection_report ORDER BY created_at DESC LIMIT 1;"
# → init|3|ep_20261010_0042

# 3. Verify via search: the hit is the merged narrative, not the old fragments
curl -s -X POST "$BASE/memory/search" \
  -H "Content-Type: application/json" \
  -d '{"query": "how many pets does Andrew have", "top_k": 5}' \
  | jq '.data.episodes[0] | {subject, episode, session_id}'
# → session_id is null on a merged narrative (the aggregation-product marker);
#   episode holds the full narrative text

See also

• how-memory-works.md — Episodes and the memory extraction pipeline
• storage_layout.md — Markdown + SQLite + LanceDB stack
• api.md — full HTTP API reference