learningcircuit--local-deep-research/docs/developing/credential-scrubbing.md

# Credential scrubbing in error/log text

`local_deep_research.security.log_sanitizer` scrubs credentials out of
exception/error strings **before** they reach a client (HTTP/SSE/JSON
responses) or the logs. It is a runtime, defense-in-depth *sanitizer* — not a
git/CI secret *scanner* (use [gitleaks](https://github.com/gitleaks/gitleaks)
for that, which this repo already runs in pre-commit/CI).

## The two layers

1. **`sanitize_error_message(text)`** — a regex first-pass over `_CREDENTIAL_PATTERNS`
   for *common credential shapes* (API-key query params, `Authorization:`/
   `x-api-key:` headers, `user:pass@host`, and well-known token prefixes:
   `sk-`/`pk-`, Google `AIza`/`ya29.`, GitHub `ghp_`/`github_pat_`, AWS
   `AKIA`/`ASIA`, Slack `xox*-`, JWTs). `sanitize_error_for_client()` composes
   it with control-char stripping + length capping.
2. **`redact_secrets(text, *known_literals)`** — the **backstop**. When you
   hold the actual secret value (e.g. the configured API key), pass it here so
   it is scrubbed regardless of shape. This is the real guarantee for
   *known* secrets; the regex layer is best-effort for *unknown* ones.

Always pair them ("dual-scrub") on any path that surfaces secret-adjacent
text — via **`scrub_error(error, *known_literals)`** (`security.log_sanitizer`),
which composes both passes plus the defensive guards (a raising `__str__`
or a non-string secret cannot crash the `except` handler). Engine
subclasses use `self._scrub_error(error)`, which resolves the engine's
`_secret_attrs` and delegates to the same helper. Don't hand-inline the
two calls: per-site copies are how scrub passes drift apart.

## Design constraints (why it is curated, not exhaustive)

This runs on **arbitrary, possibly attacker-influenced** strings at runtime, so:

- **No ReDoS.** Every pattern must scale linearly — prefer prefix-anchored,
  single-quantifier regexes; avoid nested quantifiers / ambiguous alternations.
  Spot-check new patterns on a 200k-char adversarial input.
- **Over-redaction is the safe failure; under-redaction (a leak) is not** —
  but gratuitous over-redaction harms log readability, so patterns are
  anchored/length-floored to avoid eating prose.
- Keep the set **small and high-signal**. Matching gitleaks' full 100+ rules
  at runtime is overkill and raises false positives; the `redact_secrets`
  backstop covers the long tail for known secrets.

## Keeping the patterns current

The prefix regexes mirror the **canonical, actively-maintained gitleaks
ruleset** — gitleaks' own upstream `config/gitleaks.toml`
(<https://github.com/gitleaks/gitleaks>), **not** this repo's root
`.gitleaks.toml` (which only configures the gitleaks *scan* run in
pre-commit/CI). To refresh when a provider introduces a new token format:

1. Find the rule in gitleaks' upstream `config/gitleaks.toml` (or detect-secrets).
2. Adapt the regex to a prefix-anchored, ReDoS-safe form here; redact to
   `[REDACTED_KEY]`.
3. Add a positive (redacts) **and** a negative (prose not over-redacted) case
   to `tests/security/test_log_sanitizer.py`.
4. Run `pytest tests/security/` and a 200k-char ReDoS spot-check.

A periodic (e.g. quarterly) glance at gitleaks' changelog for new
widely-used token prefixes is enough — this layer only needs the formats that
plausibly appear in *this app's* error/log text.