esengine--deepseek-reasonix/docs/SESSION_MEMORY_RETRIEVAL.md

Session History and Synthesis Memory Retrieval

This document describes the lightweight retrieval layer added for session history and saved memories. It is the implementation note behind the history and memory tools, the archive-on-forget behavior, and the fresh human approval gate for agent-written memory.

Goals

• Bring useful past-session context back without injecting dynamic history into the stable system prompt.
• Keep Reasonix cache-first: stable prompt bytes stay stable across turns, while history and saved facts are fetched on demand.
• Avoid a heavy retrieval dependency. The implementation is pure Go and does not introduce SQLite, CGO, a vector database, or an embedding model.
• Make memory trustworthy. Agent-initiated memory writes and archives must be visible to the user and approved every time.
• Preserve traceability. A wrong memory should stop affecting the agent, but the removed document should remain inspectable.

Non-Goals

• This is not semantic embedding search. It is lexical retrieval with BM25, tuned for code, commands, error phrases, filenames, and explicit decisions.
• This does not auto-summarize every session into memory. The memory layer is a synthesis cache: only stable conclusions that the user approves become saved documents.
• Archived memories are not active knowledge. They exist for audit and recovery, not for recall.

Retrieval Core

internal/retrieval contains the shared retrieval primitives:

• tokenization for Latin words and CJK runes;
• document-frequency and BM25 scoring;
• compact snippets around query terms;
• KeepTopRelativeScore, which keeps the best hit but trims weak trailing hits below a relative score floor.

The relative score floor is intentionally small (0.15) and applied after sorting. It prevents common-word-only matches from crowding out the useful hit, while still preserving multiple close matches when they are genuinely relevant.

Session History Tool

The history tool is read-only and lives in internal/history.

It supports two operations:

• search: rank saved session records by BM25.
• around: read a bounded transcript window around a returned hit.

Search input can be scoped:

• project: current session directory only.
• global: current session directory, user-global session directory, and compacted-history archive directory.

Search indexes these record kinds by default:

• user text;
• assistant text;
• tool inputs;
• tool errors.

Normal tool output is excluded by default because it can be large and noisy. It can be requested explicitly with kind=["tool_output"], optionally filtered by tool_name.

around enforces path confinement. It only accepts paths under the configured session or archive roots, so a model cannot use the tool as a general file reader.

When search returns no hits, the tool explicitly tells the agent that zero results are not proof that an event never happened. It suggests retrying with rarer terms, widening scope, or including tool output only when needed.

Saved Memory Recall Tool

The memory tool is read-only and lives in internal/memory/recall.go.

It supports:

• search: BM25 over active memory files.
• read: return one full active memory by name.
• list: show the active memory index, optionally filtered by type.

Only active memories from the project memory store participate. Archived memory files are excluded from search, read, and list.

The searchable text combines:

• slug/name;
• title;
• normalized type;
• description;
• body.

This makes short user-facing descriptions useful, while still allowing recall by the detailed body when the agent knows a rare phrase from the saved fact.

Memory as Synthesis Cache

Reasonix treats saved memory as a synthesis cache rather than as a raw transcript cache.

The intended workflow is:

1. The agent searches history or memory when it needs old context.
2. If retrieval produces a stable reusable conclusion, the agent proposes a remember write.
3. The user reviews and approves or denies the write.
4. Future sessions can reuse the saved document directly.

This avoids repeatedly paying retrieval cost for the same stable conclusion, while keeping the saved set small and auditable.

Desktop Candidate Suggestions

The desktop Memory page can scan recent local sessions and produce draft candidates:

• memory candidates from explicit long-lived preferences, rules, or project conventions in recent user turns;
• skill candidates from repeated workflow categories across recent sessions.

This is intentionally a suggestion layer, not an automatic writer:

• scanning can be run manually from the Memory page. Users may also enable a desktop UI preference that scans automatically when the Suggestions tab opens;
• candidates show their proposed body plus short evidence snippets before any write;
• accepting a memory candidate writes through the controller's active memory path, so the current session gets the same transient turn-tail update as a remember write;
• accepting a skill candidate writes through the normal skill store, preserving skill name validation, scope handling, and no-overwrite behavior.

No candidate scan changes the stable system prompt or provider-visible tool schema. Saved memories and created skills become part of the stable prefix only through the existing next-session discovery path.

Archive-on-Forget

forget no longer permanently deletes the memory file. It removes the memory from the active index and moves the file into .archive/ with a timestamped filename:

.archive/<UTC timestamp>-<name>.md

The active store and recall tool ignore archived files. Local management surfaces still expose them for traceability:

• /memory;
• CLI/TUI memory views;
• desktop memory panel.

This is important because an incorrect memory can be more disruptive than no memory, but a hard delete makes it difficult to audit how the agent reached a bad conclusion.

Human Approval Contract

Agent-initiated remember and forget calls require a fresh approval every time.

The controller treats these tools like plan approval:

• Auto approval and YOLO/full-access mode do not bypass them.
• Guardian/safety review cannot allow them on the user's behalf.
• Session grants and persistent allow rules are not created for them.
• Pending memory approvals are not drained when the user toggles auto approval.
• Non-interactive headless runs and sub-agents refuse them instead of treating Ask as autonomous allow.

The approval subject is generated from the tool arguments before the ApprovalRequest event is emitted:

• remember shows a compact preview of the name/title, normalized type, description, and body.
• forget shows the memory name being archived.

External notification hooks only receive the tool name, not the memory body, because notification channels may be less private than the local UI.

User-initiated memory edits in the desktop panel or CLI remain direct user actions and do not go through the agent approval prompt.

Boot Wiring

internal/boot registers the tools in the shared registry:

• history;
• memory;
• remember;
• forget.

The saved memory index still folds into the system prompt once at session start, after the base prompt. This preserves the cache-first prefix contract. Mid-session memory changes are injected only as transient turn-tail notes and become part of the stable prefix on the next session.

UI and CLI Surfaces

Local management surfaces distinguish active and archived memory:

• Active memories can be searched, read, and used by the agent.
• Archived memories are read-only audit entries.
• Candidate suggestions are drafts until the user confirms them.

The desktop Memory() payload always returns non-nil arrays for docs, facts, archives, and scopes. This is a Wails JSON contract: nil Go slices encode as null, while the frontend expects arrays for .map and .length.

Test Coverage

The change is covered across layers:

• retrieval scoring, snippets, tokenizer behavior, and relative-score trimming;
• history search, global/archive scope, tool input/error indexing, common-word-noise trimming, path confinement, and around;
• memory search/read/list, type filtering, 0-result fallback guidance, archived memory exclusion, and validation;
• archive-on-forget file movement, index updates, timestamp parsing, ordering, and read-only file repair;
• controller approval behavior under ask/auto/YOLO, including fresh approval and approval-preview visibility;
• boot-level tool registration and real model tool-call execution;
• desktop Memory() payload shape for active and archived facts;
• desktop memory/skill candidate generation, confirmation writes, and non-nil suggestion arrays;
• frontend CSS and TypeScript checks with generated Wails bindings.

Operational Notes

• Prefer distinctive search terms: function names, command fragments, error text, ticket IDs, file names, and decision keywords.
• Use history when the original wording or tool output matters.
• Use memory when looking for approved, stable conclusions.
• Archive wrong facts instead of overwriting them when the old fact should no longer influence the agent and should remain traceable.