learningcircuit--local-deep-research/docs/decisions/0007-egress-two-axis-data-classification.md

ADR-0007: Restructure egress around a two-axis data-classification model

Date: 2026-06-28 Status: Accepted (incremental rollout)

The model at a glance

Every model a run touches (search engine, collection / store, LLM, embeddings) is labelled on two orthogonal axes — Sensitivity (of the data it sources) and Exposure (of the sink it is). The matrix reads: a component with this Sensitivity (row) and this Exposure (column) is treated as follows.

	Contained sink	Exposing sink
Non-sensitive source	Combines with anything — public collection, local Ollama	Only alongside non-sensitive sources — public web engine, cloud LLM
Sensitive source	Only with other contained sources; never an exposing sink — private collection + local LLM	By itself only, with contained (local) inference — remote monitored store of private data

The rule in one line: a run may never let a sensitive source reach an exposing sink — unless it is explicitly switched to Permissive mode (which warns but never blocks). The rest of this document explains why this is the right model, how each component is classified, and how we roll it out.

Out of scope: the query text itself

This model classifies sources and sinks — not the question the user types. A run's query is sent verbatim to whichever search sinks the run uses, so a sensitive question typed against an exposing engine leaves the machine regardless of how the sources are labelled. The guardrail cannot inspect or sanitise query intent; that stays the user's responsibility. The UI must say so plainly — we can't protect the content of your questions, so choose sources appropriate to how sensitive your question is.

Context

The egress guardrail (src/local_deep_research/security/egress/, see its README.md) currently expresses policy on a single axis: a per-run policy.egress_scope enum (adaptive / both / public_only / private_only / strict), refined by a per-collection is_public flag and two require_local inference toggles.

This works for the common cases, but it collapses two genuinely independent properties into one word, and that conflation is now blocking us.

Problem 1 — sensitivity and exposure are different things, fused into one word

"public ↔ private" is treated as a single spectrum. Two distinct questions hide inside it:

• Is the data sensitive? — a property of a source (a collection, a document store).
• Does the destination expose data outward? — a property of a sink (a web search engine you send a query to; a cloud LLM you send chunks to).

A collection marked is_public is the clearest symptom. The word implies the data is published, but a collection is always a local store — searching it never pushes its contents to a search engine. What is_public actually authorizes is cloud inference on that data. "Public" overstates the exposure axis to describe the sensitivity axis.

Problem 2 — dual-risk sources can't be expressed

Elasticsearch and Paperless break the single axis outright. An Elasticsearch instance can be:

• a sensitive data store (your private corpus) → must not be combined with exposing sinks, and/or
• a monitored / external service (a sink that itself exposes the queries you send it).

These are opposite risks, and a single is_local / is_public flag can only pick one. Today the only safety for them is the asymmetric URL fail-up. We genuinely "cannot guarantee anything" for these engines under the current model, so for now they should not be auto-combined with other sources.

Problem 3 — both is a silent blanket

both overrides every per-source classification at once, with no visual or conceptual signal that protection has been dropped. It is the muddy middle between "classify each source" and "turn the policy off."

This is a known model

The two-axis framing is the standard Data Loss Prevention (DLP) / information-flow-control model: classify data by sensitivity, classify destinations by trust / exposure, forbid sensitive→exposing flows. The enforcement-mode vocabulary is SELinux's (enforcing / permissive / disabled). The egress package already borrows XACML / zero-trust PDP–PEP vocabulary, so this is a continuation, not a foreign import.

Decision

Re-base the egress model on two orthogonal labels:

Axis	Applies to	Values	Question
Sensitivity	sources	sensitive / non-sensitive	Must this data never leave the machine?
Exposure	sinks	exposing / contained	Does sending data here put it off-machine?

Some components play both roles — a search engine is a sink for your query and a source of results; Elasticsearch is a source of data and a sink for your query. Each gets a label on every axis it participates in. This is exactly what the single-axis model cannot represent and is the crux of the change.

Core invariant: sensitive data must not flow to an exposing sink — unless the run is explicitly switched to the escape-hatch mode.

Every model carries both labels — classification is per-component

The classification is computed per component, and "component" means every model the run touches: each search engine, each collection / store, the LLM, and the embeddings model. A single function — classify(component, settings) -> (Sensitivity, Exposure) — becomes the one home for logic scattered today across _engine_bucket, the _CLOUD_LLM_PROVIDERS set, the per-collection is_public lookup, and the URL fail-up.

Component (example)	Sensitivity	Exposure
Public collection / public web results	non-sensitive	contained
Private collection (default)	sensitive	contained
Public web / academic engine (Google, arXiv)	non-sensitive	exposing (query egress)
Cloud LLM / embeddings (Anthropic, OpenAI)	non-sensitive	exposing
Local LLM / embeddings (Ollama, sentence-transformers)	non-sensitive	contained
Paperless / Elasticsearch — local, private	sensitive	contained
Elasticsearch — remote / monitored, private	sensitive	exposing

The four quadrants — the combination rule

A run touches a set of components. Whether the set is permitted follows directly from the two labels:

#	A component that is…	may be combined with	Example
1	non-sensitive, contained	anything	public collection; local Ollama
2	sensitive, contained	other contained sources only (sensitive or not) — no exposing sink	private collection + local LLM
3	non-sensitive, exposing	non-sensitive sources only	cloud LLM; public web engine
4	sensitive + exposing	the only sensitive source (non-sensitive contained companions are fine), with contained (local) inference	remote monitored store holding private data

Underlying invariant: a run may not let a sensitive source reach an exposing sink. Because an agentic run can turn one engine's results into another engine's query (the LangGraph silent-expansion class already in our threat model), this is enforced over the whole run's component set, not per call. Equivalent phrasing: a run must be either all-non-sensitive or free of exposing sinks — with a lone quadrant-4 component allowed to run as the only sensitive source (non-sensitive contained companions are still fine), since returning its own data to itself is not a new leak and its inference is forced contained.

The escape hatch (Permissive / "Unprotected") means "suspend this invariant for this run, keep warning."

Enforcement modes (SELinux naming)

• Enforcing — current behaviour: the invariant is enforced; violating engines/sinks are blocked.
• Permissive ("Unprotected") — the escape hatch: the policy is still evaluated so the warning banners fire, but nothing is blocked. Replaces both, with an honest, loud, light-red UI. (User-facing label is an open question — "Unprotected" / "Unrestricted" / "Permissive".)

Mapping the current model onto the two axes (migration, not rewrite)

• collection is_public=False → Sensitivity sensitive; is_public=True → non-sensitive (relabel away from "public" — it never publishes).
• public web / academic engines → Exposure exposing.
• cloud LLM / cloud embeddings → Exposure exposing sinks; require_local_* = "forbid exposing inference sinks."
• local collections / local LLM (Ollama) → Exposure contained.
• scopes re-expressed: private_only ≈ "no exposing sinks"; public_only ≈ "exposing sinks allowed, sensitive sources excluded"; strict ≈ single source; adaptive ≈ infer the run's posture from the primary.
• both → removed, superseded by per-source classification + Permissive mode.
• Elasticsearch / Paperless → declared sensitive + contained by default (usable with other contained / local sources — quadrant 2), with the URL fail-up flipping exposure to exposing (quadrant 4) when the configured endpoint resolves to a public host. A per-destination trust entry (policy.trusted_search_engines) can re-contain a self-hosted instance that happens to sit on a public hostname.

Per-destination trust (the "I trust Anthropic" case)

A user-managed override that re-labels a specific sink contained (e.g. a zero-retention Anthropic endpoint, a self-hosted Paperless on a public hostname). This is the Exposure-axis counterpart of promoting a collection on the Sensitivity axis. Out of scope for the first steps; recorded here as its principled home.

Consequences

Positive

• One coherent model that matches industry DLP / IFC vocabulary.
• Dual-risk sources (Elasticsearch / Paperless) become expressible.
• Honest labels: promoting a collection is clearly about inference authorization, not publication.
• The escape hatch is explicit and visible, not a silent scope.

Negative / risks

• The egress guardrail is security-critical and has already been through two adversarial review rounds. A big-bang rewrite is high blast radius.
• Every engine / source / sink needs labels on both axes; some are genuinely ambiguous.

Therefore: rewrite incrementally, core-first, behind a test net. The model is the chosen direction, but the guardrail is security-critical and has already survived two adversarial review rounds — so it lands in stages that each keep the egress test suite green:

A. Classification core (security/egress/classification.py): the Sensitivity / Exposure types, classify(component, settings), and evaluate_run(components) implementing the four-quadrant rule — pure, fully unit-tested against the truth table above, not yet wired in (zero behaviour change). B. Wire the PEPs to the core; re-express the existing scopes in its terms; keep behaviour parity (existing egress tests stay green) while the new capabilities (quadrant 4, dual-risk sources) become reachable. C. UI: surface the two axes, add Permissive / "Unprotected" mode, remove both, make collection labels honest. D. Per-destination trust (the Exposure override — the "trust Anthropic" case) and explicit classification for Elasticsearch / Paperless.

Status (PR #4882). All four stages are implemented.

• A / B — classification core (classification.py) + resolver (run_classification.py) + explicit per-element labels on every engine and provider.
• C — the UNPROTECTED escape hatch (with the SSRF / cloud-metadata invariant preserved after the hatch gate), both fully retired — removed from the selector, existing rows migrated to adaptive by migration 0019, and any residual value (env var / queued snapshot / un-migrated DB) coerced to adaptive at read time; EgressScope.BOTH survives only as the internal resolution result for an unclassifiable ADAPTIVE primary — a non-dismissible "protection disabled" banner, and the enforcement flip: the run-start precheck rejects a two-axis denial as defense-in-depth over the scope PEPs (UNPROTECTED evaluates permissive; an uncomputable decision fails closed — a denying audit_error — not open). This runs at the web /api/start_research precheck and at the shared worker chokepoint in run_research_process, so follow-up / chat / queue runs are covered too; only CLI / programmatic callers that bypass both get the scope PEPs alone (which still force local inference under private_only / adaptive-private). The precheck classifies the primary engine plus the inference providers; widening to the full resolved engine set is a follow-up (see residuals).
• D — per-destination trust (policy.trusted_inference_providers / policy.trusted_search_engines) relaxing a trusted off-machine sink to contained, a trust banner, and honest collection-label copy.

Before it enforces in a shipped release it must pass a fresh adversarial review — the enforcement flip is a behaviour change to a security boundary.

Known residuals (tracked in issue #4951)

Four rounds of adversarial review established that the run-start audit is best-effort defense-in-depth, not a completeness guarantee: it re-derives each sink's classification from settings and can diverge from what the run actually connects to. The default adaptive config is protected by the scope PEPs (a private primary forces local inference), and retiring both removes the scope where the audit was the sole guard. These edges remain and are tracked as issues rather than blockers:

• Search axis — Elasticsearch cloud_id. The exposure fail-up inspects hosts, not cloud_id; a sensitive ES store reachable via cloud_id on a permissive scope is classified contained. The engine's own _cloud_id_forbidden_by_scope covers private_only / strict.
• Full engine set / agentic expansion. Enforcement (audit and the scope PEPs' local-inference coupling) keys on the run's primary engine; a non-primary sensitive engine pulled in mid-run by expansion is not reflected back into require_local.
• Dual-risk store on a public host. A self-hosted store whose URL resolves public classifies PUBLIC_ONLY, so the scope-level local-inference coupling does not fire (the run-start audit still flags it on web paths).
• Name-keyed trust drift. trusted_inference_providers is keyed by provider name, not the vetted endpoint URL, and is honored only in the audit, not the boundary PEP.
• LibraryRAGService direct construction resolves its primary from the global search.tool, so a direct (non-factory) construction can miss the local-embeddings coupling.
• Query text is out of scope by design (see above).

Open questions — resolution

1. Deferred. User-facing axis vocabulary. Internally settled as Sensitivity{sensitive, non_sensitive} / Exposure{contained, exposing}. The two axes are not surfaced as independent UI controls yet — they are expressed through the existing egress-scope selector, the per-collection public flag, and the trust lists. A dedicated two-axis UI is a follow-up.
2. Resolved — Unprotected. Setting value / label / EgressScope.UNPROTECTED (the internal enforcement mode is separately Mode.PERMISSIVE).
3. Resolved. Elasticsearch / Paperless default to sensitive + contained; the URL fail-up flips exposure to exposing (quadrant 4) on a public endpoint; policy.trusted_search_engines is the explicit override.
4. Resolved — no. The aggregate library is always sensitive (_resolve_collection_is_public hard-returns private for it).
5. Resolved — settings lists. policy.trusted_inference_providers / policy.trusted_search_engines (JSON-list settings, mirroring allowed_local_hostnames); no dedicated table.

References

• Egress package: src/local_deep_research/security/egress/README.md
• SELinux modes — enforcing / permissive / disabled
• DLP: data-classification (sensitivity labels) + destination-trust / egress controls (information-flow control)