learningcircuit--local-deep-research/docs/developing/credential-scrubbing.md

Credential scrubbing in error/log text

local_deep_research.security.log_sanitizer scrubs credentials out of exception/error strings before they reach a client (HTTP/SSE/JSON responses) or the logs. It is a runtime, defense-in-depth sanitizer — not a git/CI secret scanner (use gitleaks for that, which this repo already runs in pre-commit/CI).

The two layers

1. sanitize_error_message(text) — a regex first-pass over _CREDENTIAL_PATTERNS for common credential shapes (API-key query params, Authorization:/ x-api-key: headers, user:pass@host, and well-known token prefixes: sk-/pk-, Google AIza/ya29., GitHub ghp_/github_pat_, AWS AKIA/ASIA, Slack xox*-, JWTs). sanitize_error_for_client() composes it with control-char stripping + length capping.
2. redact_secrets(text, *known_literals) — the backstop. When you hold the actual secret value (e.g. the configured API key), pass it here so it is scrubbed regardless of shape. This is the real guarantee for known secrets; the regex layer is best-effort for unknown ones.

Always pair them ("dual-scrub") on any path that surfaces secret-adjacent text — via scrub_error(error, *known_literals) (security.log_sanitizer), which composes both passes plus the defensive guards (a raising __str__ or a non-string secret cannot crash the except handler). Engine subclasses use self._scrub_error(error), which resolves the engine's _secret_attrs and delegates to the same helper. Don't hand-inline the two calls: per-site copies are how scrub passes drift apart.

Design constraints (why it is curated, not exhaustive)

This runs on arbitrary, possibly attacker-influenced strings at runtime, so:

• No ReDoS. Every pattern must scale linearly — prefer prefix-anchored, single-quantifier regexes; avoid nested quantifiers / ambiguous alternations. Spot-check new patterns on a 200k-char adversarial input.
• Over-redaction is the safe failure; under-redaction (a leak) is not — but gratuitous over-redaction harms log readability, so patterns are anchored/length-floored to avoid eating prose.
• Keep the set small and high-signal. Matching gitleaks' full 100+ rules at runtime is overkill and raises false positives; the redact_secrets backstop covers the long tail for known secrets.

Keeping the patterns current

The prefix regexes mirror the canonical, actively-maintained gitleaks ruleset — gitleaks' own upstream config/gitleaks.toml (https://github.com/gitleaks/gitleaks), not this repo's root .gitleaks.toml (which only configures the gitleaks scan run in pre-commit/CI). To refresh when a provider introduces a new token format:

1. Find the rule in gitleaks' upstream config/gitleaks.toml (or detect-secrets).
2. Adapt the regex to a prefix-anchored, ReDoS-safe form here; redact to [REDACTED_KEY].
3. Add a positive (redacts) and a negative (prose not over-redacted) case to tests/security/test_log_sanitizer.py.
4. Run pytest tests/security/ and a 200k-char ReDoS spot-check.

A periodic (e.g. quarterly) glance at gitleaks' changelog for new widely-used token prefixes is enough — this layer only needs the formats that plausibly appear in this app's error/log text.