wehub-resource-sync
09e9f3545f
CodeQL / Analyze (push) Waiting to run
dependency-audit / pip-audit (push) Waiting to run
Test / Code Quality (push) Waiting to run
Test / Test (macos-latest, Python 3.10) (push) Blocked by required conditions
Test / Test (macos-latest, Python 3.11) (push) Blocked by required conditions
Test / Test (macos-latest, Python 3.12) (push) Blocked by required conditions
Test / Test (macos-latest, Python 3.13) (push) Blocked by required conditions
Test / Test (macos-latest, Python 3.14) (push) Blocked by required conditions
Test / Test (ubuntu-latest, Python 3.10) (push) Blocked by required conditions
Test / Test (ubuntu-latest, Python 3.11) (push) Blocked by required conditions
Test / Test (ubuntu-latest, Python 3.12) (push) Blocked by required conditions
Test / Test (ubuntu-latest, Python 3.13) (push) Blocked by required conditions
Test / Test (ubuntu-latest, Python 3.14) (push) Blocked by required conditions
Test / Test (windows-latest, Python 3.10) (push) Blocked by required conditions
Test / Test (windows-latest, Python 3.11) (push) Blocked by required conditions
Test / Test (windows-latest, Python 3.12) (push) Blocked by required conditions
Test / Test (windows-latest, Python 3.13) (push) Blocked by required conditions
Test / Test (windows-latest, Python 3.14) (push) Blocked by required conditions
1605 lines
80 KiB
Python
1605 lines
80 KiB
Python
"""Canonical registry of cassette-mutating utilities.
|
|
|
|
This module is the single source of truth for what counts as sensitive in a
|
|
recorded HTTP cassette and for cassette-byte-count surgery. It exports two
|
|
complementary halves:
|
|
|
|
1. **Sanitization registry.** A canonical
|
|
list of regex (pattern, replacement) pairs covering Google session
|
|
cookies, ``__Secure-*`` / ``__Host-*`` cookies, WIZ_global_data token
|
|
fields, email addresses, and Playwright ``storage_state`` cookie objects;
|
|
a single ``scrub_string`` entry point that applies them; and an
|
|
``is_clean`` validator that judges cookie-value cleanliness via exact-
|
|
match membership in ``SCRUB_PLACEHOLDERS`` (closing a previous
|
|
"starts with S" character-class hole). Before this consolidation the
|
|
same patterns lived as an inline ``SENSITIVE_PATTERNS`` list in
|
|
:mod:`tests.vcr_config` and were duplicated piecemeal in the former
|
|
``tests/check_cassettes_clean.sh`` shell guard — that drift risk is what motivated
|
|
the consolidation.
|
|
|
|
2. **Chunked-response byte-count re-derivation.** The
|
|
:func:`recompute_chunk_prefix` helper walks an XSSI-framed batchexecute
|
|
body and rewrites every digit-only ``<count>`` header to match the actual
|
|
byte-length of the immediately-following payload line. After scrubbing
|
|
replaces a 21-char user ID with the 17-char ``SCRUBBED_USER_ID``
|
|
placeholder the advertised count no longer matches the payload, so this
|
|
helper runs as a second pass inside :func:`tests.vcr_config.scrub_response`
|
|
to keep cassettes self-consistent and avoid tripping the decoder's
|
|
byte-count-mismatch DEBUG log during replay.
|
|
|
|
Why both halves live here, not split into two modules:
|
|
|
|
- ``vcr_config.py`` is loaded for every VCR-decorated test, but its public
|
|
surface is intentionally narrow (the VCR object + matchers). Scrub-time
|
|
string surgery is a separate concern and benefits from being importable
|
|
on its own (the bulk re-scrub script in ``scripts/`` imports both
|
|
``scrub_string`` AND ``recompute_chunk_prefix`` directly).
|
|
- Decoder tolerance behavior in ``src/notebooklm/rpc/decoder.py`` (still
|
|
parses the JSON on byte-count mismatch, now logging at DEBUG rather
|
|
than WARNING — see #669) is what makes the recompute pass optional for
|
|
correctness; these helpers exist so cassettes stay self-consistent for
|
|
shape-lint and don't add log noise during replay, not to harden the
|
|
decoder against drift in production responses.
|
|
|
|
Exports
|
|
-------
|
|
- :data:`SESSION_COOKIES` standard Google session cookie names
|
|
- :data:`SECURE_COOKIES` ``__Secure-*`` cookie names (caught by umbrella)
|
|
- :data:`HOST_COOKIES` ``__Host-*`` cookie names (caught by umbrella)
|
|
- :data:`OPTIONAL_COOKIES` non-essential cookies surfaced for completeness
|
|
- :data:`EMAIL_PROVIDERS` provider domains we redact in emails
|
|
- :data:`SCRUB_PLACEHOLDERS` exact-match allowlist of expected sentinels
|
|
- :data:`DISPLAY_NAME_FALSE_POSITIVES` two-Cap-word strings to NEVER scrub
|
|
- :data:`SENSITIVE_PATTERNS` ordered (regex, replacement) registry
|
|
- :func:`scrub_string` single sanitization entry point
|
|
- :func:`is_clean` validator returning ``(ok, leaks)``
|
|
- :func:`find_credential_leaks` high-severity-shape-only subset (fixture-safe)
|
|
- :func:`find_high_entropy_leaks` field-agnostic novel-token entropy backstop
|
|
- :func:`recompute_chunk_prefix` XSSI byte-count re-derivation
|
|
|
|
Necessary-not-sufficient boundary (issue #1382)
|
|
-----------------------------------------------
|
|
The name-anchored / known-shape detectors below are NECESSARY but not
|
|
SUFFICIENT: a novel credential prefix, or a known secret in an un-targeted
|
|
field, passes them silently (this has bitten the guard twice — ``LSID`` and
|
|
``JrWMbf``). :func:`find_high_entropy_leaks` is the deliberate field-agnostic
|
|
complement that catches a long high-entropy base64/hex token in ANY quoted
|
|
JSON scalar. See its KNOWN-SHAPE BOUNDARY block for the residual-risk decision
|
|
this draws (it is scoped to quoted scalars on purpose; non-quoted high-entropy
|
|
text remains GitHub-secret-scanning's job). Relates to ADR-0006.
|
|
|
|
Upload + Drive token coverage
|
|
-----------------------------
|
|
The registry below extends the canonical cookie/CSRF/email coverage with
|
|
scrubbers for Google's resumable-upload and Drive integration paths:
|
|
|
|
* ``X-GUploader-UploadID`` response headers leak per-upload session tokens.
|
|
* ``upload_id=...`` query parameters echo the same token into request URLs.
|
|
* ``AONS...`` strings are Drive ACL/permission tokens emitted with file
|
|
metadata. The 20-char tail threshold avoids matching incidental
|
|
``AONS`` mentions in code or docs.
|
|
* Drive file IDs (33-44 char ``[A-Za-z0-9_-]`` strings) appear inside
|
|
``"file_id": "..."`` JSON keys and ``/drive/v3/files/<id>`` URLs. The
|
|
pattern is intentionally context-anchored so that bare 36-char UUIDs
|
|
(artifact IDs, source IDs, conversation IDs) are NOT scrubbed — those
|
|
are NotebookLM-internal identifiers and matching them would corrupt
|
|
cassette replay.
|
|
|
|
Display-name + avatar coverage
|
|
------------------------------
|
|
The registry below also covers two display/identity leak shapes that the
|
|
core structured scrubbers miss because the data is double-encoded inside a
|
|
WRB-payload JSON string:
|
|
|
|
* **Escaped JSON display-name literals.** Google's sharing RPCs emit owner
|
|
metadata as positional list elements inside a stringified WRB payload —
|
|
the display name surfaces as ``\\"First Last\\"`` rather than a
|
|
structured ``"displayName": "..."`` key. The core structured patterns
|
|
key-anchor on the outer JSON key, so they never fire on the inner
|
|
double-encoded form. The display-name pattern anchors on the
|
|
escape-quote shape ``\\"...\\"`` and carries an explicit false-positive
|
|
allowlist (font families, UI titles, artifact/notebook names produced
|
|
by the test corpus) so that legitimate two-Capitalized-word fixture
|
|
content is preserved. This false-positive list is the load-bearing
|
|
safety net — a broad
|
|
``>[A-Z][a-z]+\\s[A-Z][a-z]+<`` regex without it would corrupt source-
|
|
rename and artifact-list cassettes during replay.
|
|
* **lh3.googleusercontent.com avatar URLs.** Both the ``/a/`` and ``/ogw/``
|
|
path forms carry per-user avatar tokens. The pattern collapses the whole
|
|
URL (host + path + token, including any trailing ``=s512``-style sizing
|
|
suffix) to ``SCRUBBED_AVATAR_URL``.
|
|
|
|
Google API-key coverage
|
|
-----------------------
|
|
The NotebookLM web page embeds a Google API key in ``WIZ_global_data``. It
|
|
surfaces across several field names (``B8SWKb``, ``VqImj``, and ``JrWMbf``);
|
|
each is scrubbed by a name-anchored pattern. ``JrWMbf`` was originally missing,
|
|
so its value round-tripped unscrubbed into the interactive mind-map cassettes —
|
|
the API-key analog of the ``LSID`` cookie leak. To close that class of gap for
|
|
good, the registry also carries a field-name-agnostic catch-all
|
|
(:data:`_GOOGLE_API_KEY_PATTERN`) that collapses the canonical ``AIza`` +
|
|
35-char Google API-key shape to ``SCRUBBED_API_KEY`` wherever it appears, with a
|
|
matching ``is_clean`` detector so any unredacted key is flagged regardless of
|
|
which field carried it.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import math
|
|
import re
|
|
from collections import Counter
|
|
|
|
# =============================================================================
|
|
# Chunked-response byte-count re-derivation
|
|
# =============================================================================
|
|
|
|
# XSSI anti-hijack prefix used by Google batchexecute responses.
|
|
# Format: ")]}'" followed by two newlines, then alternating <count>\n<payload>\n
|
|
# chunks. See ``src/notebooklm/rpc/decoder.py`` for the parser.
|
|
_XSSI_PREFIX = ")]}'\n\n"
|
|
|
|
# A "chunk header" line is a line consisting of ONLY ASCII digits — that's the
|
|
# advertised byte count for the next payload line. Restricting to ASCII digits
|
|
# avoids accidentally treating a JSON payload line that happens to start with a
|
|
# digit-like character as a header. ``fullmatch`` anchors at both ends so we
|
|
# don't need explicit ``\A`` / ``\Z`` (claude-bot review on PR #554).
|
|
_CHUNK_HEADER_RE = re.compile(r"\d+")
|
|
|
|
|
|
def recompute_chunk_prefix(body: str) -> str:
|
|
"""Re-derive ``<count>`` prefixes in a chunked response body.
|
|
|
|
Google's batchexecute responses are framed as alternating header/payload
|
|
lines, optionally preceded by the XSSI ``)]}'\\n\\n`` prefix. After
|
|
scrubbing replaces strings of unequal length (e.g. a 21-char user ID with
|
|
the 17-char ``SCRUBBED_USER_ID`` placeholder), the advertised byte-count no
|
|
longer matches the actual payload length, which causes:
|
|
|
|
1. ``tests/_guardrails/test_cassette_shapes.py`` byte-count assertion failures.
|
|
2. ``decoder.py`` to emit ``Chunk at line N declares X bytes but payload is
|
|
Y bytes`` DEBUG logs during replay (the JSON is still parsed — see
|
|
``notebooklm.rpc.decoder.parse_chunked_response`` — but well-formed cassettes
|
|
shouldn't trip the log at all).
|
|
|
|
This helper walks the body, identifies every digit-only "header" line that
|
|
is immediately followed by a non-header line, and replaces the header with
|
|
the correct count for that payload. Byte count uses ``len(payload.encode(
|
|
"utf-8"))`` — matching the ``len(json_str.encode("utf-8"))`` calculation
|
|
the decoder uses (which is what the cassette shape lint validates, even
|
|
though Google's live framing appears to use a different unit; see the
|
|
Note: block on :func:`notebooklm.rpc.decoder.parse_chunked_response`).
|
|
For ASCII-only payloads (the common case for batchexecute JSON), this is
|
|
identical to ``len(payload)``, so the shape-lint character-length
|
|
assertion in ``test_cassette_shapes.py`` still passes.
|
|
|
|
Idempotent: running the helper on a body whose counts already match yields
|
|
an identical string (no spurious whitespace changes). Conservative: if the
|
|
body doesn't look like a chunked response (no digit-only header lines), it
|
|
is returned unchanged.
|
|
|
|
Args:
|
|
body: The response body as a Python ``str``. May or may not be prefixed
|
|
with the XSSI marker.
|
|
|
|
Returns:
|
|
The body with every digit-only header line replaced by the correct
|
|
byte-count for the immediately-following payload line. Trailing
|
|
newlines, the XSSI prefix, and non-header lines are preserved verbatim.
|
|
|
|
Examples:
|
|
Single-chunk body where the payload was scrubbed shorter::
|
|
|
|
>>> recompute_chunk_prefix("18\\n[[\\"longer_id_123\\"]]")
|
|
'18\\n[["longer_id_123"]]'
|
|
>>> recompute_chunk_prefix("18\\n[[\\"x\\"]]")
|
|
'7\\n[["x"]]'
|
|
|
|
XSSI-wrapped multi-chunk body::
|
|
|
|
>>> body = ")]}'\\n\\n10\\n[1,2,3]\\n20\\n[[\\"a\\"]]\\n"
|
|
>>> # After scrubbing one payload from "[1,2,3]" to "[1,2]" the
|
|
>>> # leading "10" header becomes stale; recompute_chunk_prefix
|
|
>>> # rewrites it to match the new payload length.
|
|
|
|
"""
|
|
if not body:
|
|
return body
|
|
|
|
# Preserve the XSSI prefix exactly. Splitting on it (instead of stripping a
|
|
# fixed number of characters) is robust to alternate-length prefixes if
|
|
# Google ever changes the marker — though only ``)]}'\n\n`` is observed.
|
|
if body.startswith(_XSSI_PREFIX):
|
|
prefix = _XSSI_PREFIX
|
|
remainder = body[len(_XSSI_PREFIX) :]
|
|
else:
|
|
prefix = ""
|
|
remainder = body
|
|
|
|
# Splitting on "\n" preserves a trailing empty string if ``remainder`` ends
|
|
# in "\n", which lets us reconstruct the original terminator faithfully via
|
|
# "\n".join(...).
|
|
lines = remainder.split("\n")
|
|
|
|
out: list[str] = []
|
|
i = 0
|
|
while i < len(lines):
|
|
line = lines[i]
|
|
# A header line is followed by a non-header payload line. Only rewrite
|
|
# when BOTH conditions hold — otherwise leave the line untouched. This
|
|
# protects:
|
|
# - trailing digit-only sentinels with no payload (we leave them alone
|
|
# rather than guess what payload they would have referred to)
|
|
# - JSON payloads that happen to be a single integer literal
|
|
# immediately preceded by another digit-only line (unlikely in
|
|
# practice but we'd rather be conservative)
|
|
is_header = _CHUNK_HEADER_RE.fullmatch(line) is not None
|
|
has_payload = i + 1 < len(lines) and not _CHUNK_HEADER_RE.fullmatch(lines[i + 1])
|
|
if is_header and has_payload:
|
|
payload = lines[i + 1]
|
|
new_count = len(payload.encode("utf-8"))
|
|
out.append(str(new_count))
|
|
out.append(payload)
|
|
i += 2
|
|
else:
|
|
out.append(line)
|
|
i += 1
|
|
|
|
return prefix + "\n".join(out)
|
|
|
|
|
|
# =============================================================================
|
|
# Cookie name categories
|
|
# =============================================================================
|
|
|
|
# Standard Google session cookies. These are the names whose values we scrub
|
|
# from both the ``Cookie:`` / ``Set-Cookie:`` header form AND the Playwright
|
|
# ``storage_state`` JSON form.
|
|
SESSION_COOKIES: list[str] = [
|
|
"SID",
|
|
"HSID",
|
|
"SSID",
|
|
"APISID",
|
|
"SAPISID",
|
|
"SIDCC",
|
|
"OSID",
|
|
"NID",
|
|
# ``LSID`` is the Google **login** SID cookie. Its value embeds a raw
|
|
# ``g.a000-...`` SID token — the same credential family as ``SID`` — yet it
|
|
# was historically MISSING from this list, so its value round-tripped into
|
|
# committed cassettes unscrubbed (the ``notebooks_share.yaml`` leak that
|
|
# motivated this hardening). ``LSOLH`` is the sibling login cookie that
|
|
# carries the same token shape; both are added so neither can leak again.
|
|
"LSID",
|
|
"LSOLH",
|
|
]
|
|
|
|
# ``__Secure-*`` cookies are caught by the umbrella ``__Secure-[^=]+`` pattern;
|
|
# this list is the canonical enumeration of names we expect to see in practice.
|
|
SECURE_COOKIES: list[str] = [
|
|
"__Secure-1PSID",
|
|
"__Secure-3PSID",
|
|
"__Secure-1PSIDCC",
|
|
"__Secure-3PSIDCC",
|
|
"__Secure-1PSIDTS",
|
|
"__Secure-3PSIDTS",
|
|
"__Secure-1PAPISID",
|
|
"__Secure-3PAPISID",
|
|
"__Secure-OSID",
|
|
]
|
|
|
|
# ``__Host-*`` cookies, caught by the umbrella ``__Host-[^=]+`` pattern.
|
|
HOST_COOKIES: list[str] = [
|
|
"__Host-GAPS",
|
|
]
|
|
|
|
# Optional / non-essential Google cookies. We expose the list for completeness
|
|
# but do NOT scrub their values today (they don't carry session secrets).
|
|
OPTIONAL_COOKIES: list[str] = [
|
|
"1P_JAR",
|
|
"AEC",
|
|
"CONSENT",
|
|
]
|
|
|
|
# =============================================================================
|
|
# Email provider domains we redact
|
|
# =============================================================================
|
|
|
|
EMAIL_PROVIDERS: list[str] = [
|
|
"gmail",
|
|
"googlemail",
|
|
"google",
|
|
"anthropic",
|
|
"outlook",
|
|
"hotmail",
|
|
"yahoo",
|
|
"icloud",
|
|
"protonmail",
|
|
]
|
|
|
|
# =============================================================================
|
|
# Placeholder allowlist
|
|
# =============================================================================
|
|
# These are the only string values that may appear in place of redacted secrets
|
|
# inside a committed cassette. ``is_clean`` uses this set as an exact-match
|
|
# allowlist when deciding whether a residual cookie value is a real leak — this
|
|
# replaces the legacy ``[^S"]`` character-class heuristic that missed any real
|
|
# secret starting with the letter ``S``.
|
|
SCRUB_PLACEHOLDERS: frozenset[str] = frozenset(
|
|
{
|
|
"SCRUBBED",
|
|
"SCRUBBED_CSRF",
|
|
"SCRUBBED_SESSION",
|
|
"SCRUBBED_USER_ID",
|
|
"SCRUBBED_API_KEY",
|
|
"SCRUBBED_CLIENT_ID",
|
|
"SCRUBBED_PROJECT_ID",
|
|
"SCRUBBED_EMAIL",
|
|
"SCRUBBED_NAME",
|
|
# ``SCRUBBED_EMAIL@example.com`` is the rendered form of the email
|
|
# replacement; ``is_clean`` checks the full token, so we list it too.
|
|
"SCRUBBED_EMAIL@example.com",
|
|
# URL-encoded form for ``?authuser=`` query params. The provider-
|
|
# agnostic URL detector would otherwise re-flag the canonical
|
|
# placeholder as a leak (idempotency).
|
|
"authuser=SCRUBBED_EMAIL%40example.com",
|
|
# Double-encoded form for ``%3Fauthuser%3D…`` redirect-param URLs
|
|
# (issue #1368). Same idempotency rationale as the single-encoded
|
|
# placeholder above.
|
|
"authuser%3DSCRUBBED_EMAIL%40example.com",
|
|
# upload + Drive token placeholders.
|
|
"SCRUBBED_UPLOAD_ID",
|
|
"SCRUBBED_UPLOAD_URL",
|
|
"SCRUBBED_AONS",
|
|
"SCRUBBED_DRIVE_FILE_ID",
|
|
# avatar URL placeholder (display-name + avatar scrub group).
|
|
# The display-name escaped-literal scrubber reuses the existing
|
|
# ``SCRUBBED_NAME`` sentinel so a cassette can carry just one
|
|
# canonical replacement string for human names.
|
|
"SCRUBBED_AVATAR_URL",
|
|
}
|
|
)
|
|
|
|
|
|
# =============================================================================
|
|
# Display-name false-positive allowlist
|
|
# =============================================================================
|
|
# Two-Capitalized-word strings that LOOK like human display names but are
|
|
# legitimate UI / font-family / artifact / notebook titles produced during
|
|
# E2E test runs. The escaped-display-name scrubber (section 12 below) carries
|
|
# this as a negative lookahead so these strings are NEVER replaced — that
|
|
# protection is what keeps source-rename and artifact-list cassettes from
|
|
# being corrupted during replay.
|
|
#
|
|
# This list intentionally mirrors ``DISPLAY_NAME_FALSE_POSITIVES`` in
|
|
# ``tests/_guardrails/test_cassette_shapes.py``. The two lists are NOT
|
|
# imported from each other to keep ``cassette_patterns.py`` a leaf module —
|
|
# the shape-lint module already depends on this registry, and a back-edge
|
|
# would create a cycle. New entries must be added to BOTH lists. The unit
|
|
# test ``test_display_name_false_positives_mirror_shape_lint`` asserts they
|
|
# stay in sync.
|
|
DISPLAY_NAME_FALSE_POSITIVES: frozenset[str] = frozenset(
|
|
{
|
|
# Google Sans family (font-family CSS in HTML responses).
|
|
"Google Sans",
|
|
"Google Sans Text",
|
|
"Google Sans Arabic",
|
|
"Google Sans Japanese",
|
|
"Google Sans Korean",
|
|
"Google Sans Simplified Chinese",
|
|
"Google Sans Traditional Chinese",
|
|
# Browser user-agent brand surfaced in Sec-CH-UA HTML responses.
|
|
"Microsoft Edge",
|
|
# Account UI page title (not a person's name).
|
|
"Account Information",
|
|
# Artifact / notebook titles produced by the test corpus.
|
|
"Agent Development Tutorials",
|
|
"Agent Flashcards",
|
|
"Agent Quiz",
|
|
"Slide Deck",
|
|
"Tool Use Loop",
|
|
"Claude Code",
|
|
}
|
|
)
|
|
|
|
|
|
# =============================================================================
|
|
# Pattern construction helpers
|
|
# =============================================================================
|
|
|
|
_EMAIL_PATTERN_BASE = r"[A-Za-z0-9._%+\-]+@(?:" + "|".join(EMAIL_PROVIDERS) + r")\.com"
|
|
|
|
# Single-encoded ``authuser=<local>%40<provider>`` query-param shape and its
|
|
# double-encoded ``authuser%3D<local>%40<provider>`` sibling. The double-encoded
|
|
# form arises when the email-bearing inner URL is itself the *value* of a
|
|
# ``continue=`` redirect param, so its ``?authuser=`` gets percent-encoded one
|
|
# extra level (``?``→``%3F``, ``=``→``%3D``, ``@``→``%40``) — issue #1368. Both
|
|
# anchor on the ``authuser`` separator (literal ``=`` vs encoded ``%3D``) rather
|
|
# than on the email's domain, so Workspace / corporate addresses the
|
|
# provider-list pattern misses are still caught. The shared email tail uses the
|
|
# wire-form local part (``%`` is legal because ``%2B`` etc. arrive encoded)
|
|
# followed by the encoded ``%40`` separator and a domain. Neither shape has a
|
|
# legitimate non-secret occurrence in a cassette, so collapsing to the canonical
|
|
# placeholder is safe and the detectors below treat a match as a leak by
|
|
# definition.
|
|
#
|
|
# ``AUTHUSER_EMAIL_DOUBLE_ENCODED_PATTERN`` is public (no leading underscore)
|
|
# because ``scripts/rescrub-cassettes.py`` imports it across the module boundary
|
|
# — matching the convention every other cross-imported name in this module
|
|
# (``SENSITIVE_PATTERNS``, ``SCRUB_PLACEHOLDERS``, ``find_credential_leaks`` …)
|
|
# follows. The ``_AUTHUSER_EMAIL_TAIL`` / ``_AUTHUSER_EMAIL_PATTERN`` helpers
|
|
# stay private; they are only consumed within this module.
|
|
_AUTHUSER_EMAIL_TAIL = r"[A-Za-z0-9._%+\-]+%40[A-Za-z0-9.\-]+\.[A-Za-z]{2,}"
|
|
_AUTHUSER_EMAIL_PATTERN = r"authuser=" + _AUTHUSER_EMAIL_TAIL
|
|
AUTHUSER_EMAIL_DOUBLE_ENCODED_PATTERN = r"authuser%3D" + _AUTHUSER_EMAIL_TAIL
|
|
|
|
# Negative-lookahead alternation built from the false-positive allowlist.
|
|
# Each entry is regex-escaped because some legitimate UI titles could in
|
|
# theory contain regex metacharacters (none do today, but future additions
|
|
# might). Sort by descending length so longer prefixes match before shorter
|
|
# ones — e.g. ``Google Sans Text`` must be tried before ``Google Sans``,
|
|
# otherwise the lookahead would consume only the shared prefix and the
|
|
# scrubber would proceed to clobber the longer name.
|
|
_DISPLAY_NAME_ALLOWLIST_ALT = "|".join(
|
|
re.escape(name) for name in sorted(DISPLAY_NAME_FALSE_POSITIVES, key=len, reverse=True)
|
|
)
|
|
|
|
|
|
# Catch-all Google auth-token shapes, applied as a defense-in-depth first pass
|
|
# over EVERY scrubbed string (bodies + headers + cookie values). These are the
|
|
# raw credential prefixes that ride inside session cookies, OAuth flows, and
|
|
# SIDTS rotation tokens — the ``LSID`` leak proved that a cookie-name allowlist
|
|
# alone is insufficient, so we also scrub the token shapes directly wherever
|
|
# they appear. ``is_clean`` carries the matching detector
|
|
# (``_DETECT_AUTH_TOKEN``) so the cassette guard flags any of these shapes that
|
|
# survive.
|
|
#
|
|
# Anchoring strategy per shape:
|
|
#
|
|
# * ``g\.a000-`` — anchored on the literal ``g.a000-`` prefix (note the
|
|
# REQUIRED trailing ``-``). That prefix is itself distinctive enough that
|
|
# no length floor is needed: ``g.a000-<anything>`` in a cassette is a SID
|
|
# token by construction and is never legitimate fixture content, so we
|
|
# scrub even a one-char tail. The hyphen requirement keeps the bare
|
|
# account-prefix ``g.a000`` (no token tail) from matching.
|
|
# * ``sidts-`` / ``ya29\.`` — less distinctive prefixes, so each carries a
|
|
# length floor (``{10,}`` / ``{20,}``) to avoid firing on an incidental
|
|
# short literal such as a bare ``ya29`` mention in a comment.
|
|
#
|
|
# Anything matched collapses to ``SCRUBBED`` (which contains none of the
|
|
# prefixes), so repeated passes are idempotent.
|
|
_AUTH_TOKEN_PATTERNS: list[str] = [
|
|
r"g\.a000-[A-Za-z0-9_\-]+",
|
|
r"sidts-[A-Za-z0-9_\-]{10,}",
|
|
r"ya29\.[A-Za-z0-9_\-]{20,}",
|
|
]
|
|
|
|
# Google API-key shape (``AIza`` + 35 ``[A-Za-z0-9_-]`` chars), applied as a
|
|
# defense-in-depth catch-all alongside the name-anchored WIZ_global_data
|
|
# ``B8SWKb`` / ``VqImj`` / ``JrWMbf`` scrubbers below. The name-anchored
|
|
# patterns scrub whatever a *known* API-key field carries; this shape pattern
|
|
# scrubs the key wherever it appears regardless of the surrounding field name.
|
|
# The ``JrWMbf`` leak (the NotebookLM web API key round-tripping into the
|
|
# ``generate_mind_map_interactive`` / ``mind_maps_interactive`` cassettes) is
|
|
# the API-key analog of the ``LSID`` cookie leak: a credential rode in a field
|
|
# that was not on the allowlist, so a name-anchored guard alone missed it. The
|
|
# ``AIza`` prefix + a 35-char tail is the canonical Google API-key shape and
|
|
# never occurs as legitimate fixture content, so collapsing it to the
|
|
# ``SCRUBBED_API_KEY`` sentinel is safe. ``is_clean`` carries the matching
|
|
# detector (``_DETECT_GOOGLE_API_KEY``) so any surviving raw key is flagged.
|
|
#
|
|
# The tail uses ``{35,}`` (35-OR-MORE), not an exact ``{35}``: a greedy
|
|
# open-ended quantifier consumes the WHOLE contiguous key-char run. With an
|
|
# exact ``{35}`` a longer-than-canonical key (e.g. ``AIza`` + 36 chars) would be
|
|
# scrubbed only up to its first 39 chars, leaving a trailing fragment that the
|
|
# ``SCRUBBED_API_KEY`` replacement no longer re-matches — a silent partial leak
|
|
# in an unknown field with no name-anchored backstop (gemini review on #1266).
|
|
_GOOGLE_API_KEY_PATTERN = r"AIza[0-9A-Za-z_\-]{35,}"
|
|
|
|
|
|
def _cookie_header_replacer(name: str) -> tuple[str, str]:
|
|
"""Build (regex, replacement) for a Cookie / Set-Cookie header pattern.
|
|
|
|
Uses a negative lookbehind anchor so a legitimate non-protected cookie
|
|
whose name *ends* with a protected name (e.g. ``BSID=...``) is not
|
|
accidentally scrubbed — see ``tests/unit/test_cookie_redaction.py``.
|
|
"""
|
|
return (
|
|
rf"(?<![A-Za-z0-9_-]){re.escape(name)}=[^;]+",
|
|
f"{name}=SCRUBBED",
|
|
)
|
|
|
|
|
|
# =============================================================================
|
|
# Name-AGNOSTIC cookie-header scrubbing + detection
|
|
# =============================================================================
|
|
# The name-anchored ``_cookie_header_replacer`` patterns above scrub only the
|
|
# cookie NAMES they enumerate (``SESSION_COOKIES`` + the ``__Secure-*`` /
|
|
# ``__Host-*`` umbrellas). Any cookie OUTSIDE those lists kept its real value in
|
|
# the committed cassette — confirmed for Google Analytics cookies (``_ga``,
|
|
# ``_ga_<id>``, ``_gcl_au``) and one-off cookies like ``AEC`` /
|
|
# ``SEARCH_SAMESITE``. Analytics client-ids and timestamps are low-blast-radius
|
|
# but they ARE real per-user values, so the bar is: NO cookie value anywhere in
|
|
# a committed cassette retains a real value, name-agnostically.
|
|
#
|
|
# The functions below operate on a single LOGICAL cookie-header value (the
|
|
# folded YAML scalar already joined into one string — the recorder hands
|
|
# :func:`scrub_request` / :func:`scrub_response` exactly this form, and the gate
|
|
# parses the YAML to recover it). They never run over the raw wrapped YAML text,
|
|
# so a cookie pair split across a YAML line-wrap boundary can't corrupt anything.
|
|
#
|
|
# Cookie-header value shape (request ``Cookie:``): ``name=value; name=value; …``
|
|
# — every ``;``-separated segment is a ``name=value`` cookie pair.
|
|
# Set-Cookie value shape (response ``Set-Cookie:``): ``name=value; Attr=x; Attr; …``
|
|
# — ONLY the first segment is the cookie pair; the rest are attributes
|
|
# (``Path`` / ``Domain`` / ``Expires`` / ``Secure`` / ``HttpOnly`` /
|
|
# ``SameSite`` / ``Priority`` / ``Max-Age`` / ``SameParty``) and must be
|
|
# preserved verbatim.
|
|
|
|
# Reserved Set-Cookie attribute names (lowercased). A segment after the first
|
|
# whose key is in this set is an attribute, not a cookie pair, and is left
|
|
# untouched. ``Secure`` / ``HttpOnly`` are valueless flags; the rest carry a
|
|
# value we must NOT scrub (``Path=/``, ``Domain=.google.com``, ...).
|
|
_SET_COOKIE_ATTR_NAMES: frozenset[str] = frozenset(
|
|
{
|
|
"path",
|
|
"domain",
|
|
"expires",
|
|
"max-age",
|
|
"secure",
|
|
"httponly",
|
|
"samesite",
|
|
"priority",
|
|
"sameparty",
|
|
"partitioned",
|
|
}
|
|
)
|
|
|
|
# Canonical placeholder a scrubbed cookie value collapses to. Reuses the bare
|
|
# ``SCRUBBED`` sentinel the name-anchored patterns already emit so a cassette
|
|
# carries one canonical cookie-value placeholder regardless of how the value was
|
|
# scrubbed.
|
|
_COOKIE_VALUE_PLACEHOLDER = "SCRUBBED"
|
|
|
|
|
|
def _scrub_cookie_pairs(header_value: str, *, first_pair_only: bool) -> str:
|
|
"""Replace cookie-pair VALUES in a logical cookie-header value, name-agnostic.
|
|
|
|
Every ``name=value`` pair has its value replaced with the canonical
|
|
:data:`_COOKIE_VALUE_PLACEHOLDER`, preserving the name and the surrounding
|
|
``; `` separators verbatim. Already-scrubbed values (the placeholder) pass
|
|
through unchanged, so the function is idempotent.
|
|
|
|
Args:
|
|
header_value: One logical cookie-header value, e.g.
|
|
``"SID=abc; _ga=GA1.1.x; NID=def"``.
|
|
first_pair_only: When ``True`` (the Set-Cookie case) only the FIRST
|
|
``name=value`` segment is treated as a cookie pair; every later
|
|
``;``-delimited segment is a cookie ATTRIBUTE (``Path`` / ``Domain``
|
|
/ ``Expires`` / ``Secure`` / ...) and is preserved verbatim. When
|
|
``False`` (the request ``Cookie:`` case) every segment is a pair.
|
|
|
|
Returns:
|
|
The header value with cookie values cleared. Whitespace and ``;``
|
|
layout are preserved exactly.
|
|
"""
|
|
# Split on ";" but keep the separators so the original spacing round-trips.
|
|
segments = re.split(r"(;)", header_value)
|
|
out: list[str] = []
|
|
pair_index = 0
|
|
for segment in segments:
|
|
if segment == ";":
|
|
out.append(segment)
|
|
continue
|
|
# ``leading``/``core``/``trailing`` preserve surrounding whitespace so
|
|
# ``" _ga=x"`` stays ``" _ga=SCRUBBED"`` rather than losing its space.
|
|
stripped = segment.strip()
|
|
if not stripped:
|
|
out.append(segment)
|
|
continue
|
|
leading_len = len(segment) - len(segment.lstrip())
|
|
trailing_len = len(segment) - len(segment.rstrip())
|
|
leading = segment[:leading_len]
|
|
trailing = segment[len(segment) - trailing_len :] if trailing_len else ""
|
|
core = stripped
|
|
|
|
is_pair_position = (not first_pair_only) or pair_index == 0
|
|
if "=" in core:
|
|
name, _, value = core.partition("=")
|
|
key_lower = name.strip().lower()
|
|
if first_pair_only and pair_index > 0 and key_lower in _SET_COOKIE_ATTR_NAMES:
|
|
# Set-Cookie attribute carrying a value (Path=/, Domain=...): keep.
|
|
out.append(segment)
|
|
continue
|
|
if is_pair_position:
|
|
if value == _COOKIE_VALUE_PLACEHOLDER:
|
|
out.append(segment) # idempotent: already scrubbed.
|
|
else:
|
|
out.append(f"{leading}{name}={_COOKIE_VALUE_PLACEHOLDER}{trailing}")
|
|
pair_index += 1
|
|
continue
|
|
# first_pair_only and we've passed the cookie pair: a non-reserved
|
|
# ``k=v`` attribute (rare). Leave it alone — it's not the cookie.
|
|
out.append(segment)
|
|
continue
|
|
# Valueless segment (a flag like ``Secure`` / ``HttpOnly``, or a bare
|
|
# cookie name with no value). Preserve verbatim; nothing to scrub.
|
|
out.append(segment)
|
|
if is_pair_position:
|
|
pair_index += 1
|
|
return "".join(out)
|
|
|
|
|
|
def scrub_cookie_header(header_value: str) -> str:
|
|
"""Scrub EVERY cookie value in a request ``Cookie:`` header, name-agnostic.
|
|
|
|
Each ``name=value`` pair's value is replaced with ``SCRUBBED`` while the
|
|
name and ``; `` layout are preserved. Idempotent on already-scrubbed input.
|
|
This is the name-agnostic complement to the name-anchored
|
|
``_cookie_header_replacer`` patterns — it catches cookies (``_ga``,
|
|
``_gcl_au``, ``AEC``, …) that are NOT on :data:`SESSION_COOKIES` and would
|
|
otherwise keep their real values in committed cassettes.
|
|
"""
|
|
return _scrub_cookie_pairs(header_value, first_pair_only=False)
|
|
|
|
|
|
def scrub_set_cookie(header_value: str) -> str:
|
|
"""Scrub the cookie value in a response ``Set-Cookie:`` header, name-agnostic.
|
|
|
|
Only the leading ``name=value`` cookie pair is scrubbed; trailing attributes
|
|
(``Path`` / ``Domain`` / ``Expires`` / ``Secure`` / ``HttpOnly`` /
|
|
``SameSite`` / ``Priority`` / ``Max-Age``) are preserved verbatim.
|
|
Idempotent on already-scrubbed input.
|
|
"""
|
|
return _scrub_cookie_pairs(header_value, first_pair_only=True)
|
|
|
|
|
|
def find_cookie_leaks(header_value: str, *, set_cookie: bool = False) -> list[str]:
|
|
"""Return name-agnostic cookie-value leaks in a logical cookie-header value.
|
|
|
|
Flags any ``name=value`` cookie pair whose value is not the canonical
|
|
placeholder. For a ``Set-Cookie`` value (``set_cookie=True``) only the first
|
|
pair is a cookie; trailing attributes (``Path``/``Domain``/...) are skipped.
|
|
Each returned string is a human-readable ``"Leak (...): ..."`` description.
|
|
|
|
This is the field-agnostic detector that catches the ``_ga`` class going
|
|
forward — it does NOT depend on a cookie name being on any allowlist.
|
|
"""
|
|
leaks: list[str] = []
|
|
shape = "Set-Cookie header" if set_cookie else "Cookie header"
|
|
segments = header_value.split(";")
|
|
pair_index = 0
|
|
for segment in segments:
|
|
core = segment.strip()
|
|
if not core or "=" not in core:
|
|
# Valueless flag (Secure/HttpOnly) or empty segment.
|
|
if core:
|
|
pair_index += 1
|
|
continue
|
|
name, _, value = core.partition("=")
|
|
name = name.strip()
|
|
key_lower = name.lower()
|
|
# Reserved Set-Cookie attribute names (``path`` / ``domain`` / ``expires``
|
|
# / ...) are NEVER real cookie names, so skip them unconditionally after
|
|
# the first pair. This keeps the detector correct even when the caller
|
|
# cannot tell a request ``Cookie:`` line from a ``Set-Cookie:`` line
|
|
# (the line-based ``is_clean`` path, where the header label is on a
|
|
# different physical line) — a Set-Cookie ``expires=...; path=/`` run is
|
|
# not mistaken for leaking cookies.
|
|
if pair_index > 0 and key_lower in _SET_COOKIE_ATTR_NAMES:
|
|
continue
|
|
is_pair_position = (not set_cookie) or pair_index == 0
|
|
if is_pair_position and value not in SCRUB_PLACEHOLDERS:
|
|
leaks.append(
|
|
f"Leak ({shape}): cookie {name!r} value {value!r} is not a known scrub placeholder"
|
|
)
|
|
if is_pair_position:
|
|
pair_index += 1
|
|
return leaks
|
|
|
|
|
|
# Single cookie-name alternation reused by EVERY JSON-shape cookie scrubber
|
|
# (storage_state name-first / value-first / bare-key forms). Derived from
|
|
# :data:`SESSION_COOKIES` plus the ``__Secure-*`` / ``__Host-*`` umbrellas so a
|
|
# name added to the registry (the ``LSID`` / ``LSOLH`` additions that motivated
|
|
# this hardening) is picked up by the header form AND the JSON forms in one
|
|
# place. Previously these alternations were hand-duplicated per pattern, so
|
|
# ``LSID`` would have been scrubbed in the header but only partially scrubbed in
|
|
# storage_state JSON — and ``is_clean`` (whose detector IS registry-derived via
|
|
# ``_COOKIE_NAMES_GROUP``) would then flag the residual value as a leak. Keeping
|
|
# scrubber and detector both registry-derived closes that drift.
|
|
_COOKIE_JSON_NAMES_GROUP = (
|
|
"|".join(re.escape(name) for name in SESSION_COOKIES) + r'|__Secure-[^"]+|__Host-[^"]+'
|
|
)
|
|
|
|
|
|
# =============================================================================
|
|
# Sensitive patterns
|
|
# =============================================================================
|
|
# The list is order-sensitive: earlier patterns run first. Each entry is a
|
|
# ``(regex, replacement)`` pair consumed by :func:`re.sub` in :func:`scrub_string`
|
|
# below. Most replacements are static strings; display-name and Drive-file-ID
|
|
# scrubbers use context-aware (callable) replacements where exact-match
|
|
# allowlists or surrounding context need to be consulted.
|
|
SENSITIVE_PATTERNS: list[tuple[str, str]] = [
|
|
# -------------------------------------------------------------------------
|
|
# 0. Catch-all Google auth-token shapes (defense in depth)
|
|
# -------------------------------------------------------------------------
|
|
# These run FIRST, before any cookie-name-anchored pattern, so a session
|
|
# token leaks NOTHING even when it rides inside a cookie whose name is not
|
|
# on the allowlist (the ``LSID`` leak class), inside a response/request
|
|
# BODY, or inside any header value. Each shape is a distinctive,
|
|
# high-entropy Google credential prefix with no legitimate non-secret
|
|
# occurrence in a cassette:
|
|
#
|
|
# * ``g.a000-...`` — the raw SID token embedded in SID/LSID cookie
|
|
# values and OAuth flows.
|
|
# * ``sidts-...`` — the ``__Secure-*PSIDTS`` rotation-timestamp token.
|
|
# * ``ya29....`` — Google OAuth2 access tokens.
|
|
#
|
|
# See :data:`_AUTH_TOKEN_PATTERNS` for the per-shape anchoring strategy
|
|
# (``g.a000-`` is distinctive enough to need no length floor; ``sidts-`` /
|
|
# ``ya29.`` carry floors to avoid incidental short-literal matches).
|
|
# Anything matched collapses to ``SCRUBBED`` so no fragment of the original
|
|
# token survives, and the replacement does not itself re-match (idempotent).
|
|
*((p, "SCRUBBED") for p in _AUTH_TOKEN_PATTERNS),
|
|
# Google API-key shape catch-all (defense in depth). Runs before the
|
|
# name-anchored WIZ_global_data API-key scrubbers in section 4 so a key
|
|
# leaks NOTHING even when it rides in a WIZ field that is not on the
|
|
# allowlist (the ``JrWMbf`` leak class). Collapses to ``SCRUBBED_API_KEY``,
|
|
# which does not itself re-match (idempotent).
|
|
(_GOOGLE_API_KEY_PATTERN, "SCRUBBED_API_KEY"),
|
|
# -------------------------------------------------------------------------
|
|
# 1. Cookie-header form: "Name=Value; ..."
|
|
# -------------------------------------------------------------------------
|
|
*(_cookie_header_replacer(name) for name in SESSION_COOKIES),
|
|
# ``__Secure-*`` / ``__Host-*`` umbrellas — the prefix is distinctive
|
|
# enough that no legitimate non-protected cookie shares it, so no
|
|
# lookbehind anchor is needed.
|
|
(r"(__Secure-[^=]+)=[^;]+", r"\1=SCRUBBED"),
|
|
(r"(__Host-[^=]+)=[^;]+", r"\1=SCRUBBED"),
|
|
# -------------------------------------------------------------------------
|
|
# 2. CSRF and session tokens in WIZ_global_data (HTML / JSON responses)
|
|
# -------------------------------------------------------------------------
|
|
# The value match uses the escape-aware idiom ``(?:[^"\\]|\\.)*`` (matched
|
|
# to the cookie-shape patterns below). A naive ``[^"]+`` would stop at the
|
|
# first JSON-escaped quote (``\"``) and leave the tail of a secret in the
|
|
# cassette while still producing a "SCRUBBED" prefix that ``is_clean``
|
|
# accepts as a placeholder — silently leaking the suffix.
|
|
(r'"SNlM0e"\s*:\s*"(?:[^"\\]|\\.)*"', '"SNlM0e":"SCRUBBED_CSRF"'),
|
|
(r'"FdrFJe"\s*:\s*"(?:[^"\\]|\\.)*"', '"FdrFJe":"SCRUBBED_SESSION"'),
|
|
# -------------------------------------------------------------------------
|
|
# 3. URL / form-body parameters
|
|
# -------------------------------------------------------------------------
|
|
(r"f\.sid=[^&]+", "f.sid=SCRUBBED"),
|
|
# Negative lookbehind anchors the param-name boundary so legitimate
|
|
# parameters whose names *end* in ``at`` (``flat=...``, ``rate=...``,
|
|
# ``format=...``) are not accidentally scrubbed.
|
|
(r"(?<![A-Za-z0-9_-])at=[A-Za-z0-9_-]+", "at=SCRUBBED_CSRF"),
|
|
(r'"at"\s*:\s*"(?:[^"\\]|\\.)*"', '"at":"SCRUBBED_CSRF"'),
|
|
# -------------------------------------------------------------------------
|
|
# 4. PII / IDs in WIZ_global_data
|
|
# -------------------------------------------------------------------------
|
|
(r'"oPEP7c"\s*:\s*"(?:[^"\\]|\\.)*"', '"oPEP7c":"SCRUBBED_EMAIL"'),
|
|
(r'"S06Grb"\s*:\s*"(?:[^"\\]|\\.)*"', '"S06Grb":"SCRUBBED_USER_ID"'),
|
|
(r'"W3Yyqf"\s*:\s*"(?:[^"\\]|\\.)*"', '"W3Yyqf":"SCRUBBED_USER_ID"'),
|
|
(r'"qDCSke"\s*:\s*"(?:[^"\\]|\\.)*"', '"qDCSke":"SCRUBBED_USER_ID"'),
|
|
(r'"B8SWKb"\s*:\s*"(?:[^"\\]|\\.)*"', '"B8SWKb":"SCRUBBED_API_KEY"'),
|
|
(r'"VqImj"\s*:\s*"(?:[^"\\]|\\.)*"', '"VqImj":"SCRUBBED_API_KEY"'),
|
|
# ``JrWMbf`` is the NotebookLM web API key embedded in WIZ_global_data. It
|
|
# was missing from this list, so its value round-tripped unscrubbed into the
|
|
# interactive mind-map cassettes (the leak this scrubber closes). Sibling of
|
|
# the ``B8SWKb`` / ``VqImj`` API-key fields above.
|
|
(r'"JrWMbf"\s*:\s*"(?:[^"\\]|\\.)*"', '"JrWMbf":"SCRUBBED_API_KEY"'),
|
|
(r'"QGcrse"\s*:\s*"(?:[^"\\]|\\.)*"', '"QGcrse":"SCRUBBED_CLIENT_ID"'),
|
|
(r'"iQJtYd"\s*:\s*"(?:[^"\\]|\\.)*"', '"iQJtYd":"SCRUBBED_PROJECT_ID"'),
|
|
# -------------------------------------------------------------------------
|
|
# 5. Email addresses
|
|
# -------------------------------------------------------------------------
|
|
# JSON-quoted form. The replacement embeds ``@example.com`` so a second
|
|
# scrub pass on already-scrubbed content is a no-op (idempotent).
|
|
(f'"{_EMAIL_PATTERN_BASE}"', '"SCRUBBED_EMAIL@example.com"'),
|
|
# ``authuser=<email>`` query-param form. The client appends this to
|
|
# every batchexecute URL whenever ``account_email`` is set, so request
|
|
# URIs would otherwise leak the maintainer's email. Anchoring on
|
|
# ``authuser=`` (not the email's domain) scrubs Workspace / corporate
|
|
# addresses the provider-list pattern misses, with no false-positive
|
|
# risk elsewhere. The replacement keeps the ``%40`` shape so VCR
|
|
# matchers still see a well-formed value on replay.
|
|
(_AUTHUSER_EMAIL_PATTERN, "authuser=SCRUBBED_EMAIL%40example.com"),
|
|
# Double-encoded ``authuser%3D<email>`` form (issue #1368). When the
|
|
# email-bearing inner URL is the *value* of a ``continue=`` redirect param
|
|
# (Google account-menu ``SignOutOptions`` / ``brandaccounts`` links), its
|
|
# ``?authuser=`` is percent-encoded one extra level, so the literal-``=``
|
|
# pattern above never fires. Mirrors the single-encoded rule with the
|
|
# canonical ``authuser%3DSCRUBBED_EMAIL%40example.com`` placeholder, which
|
|
# does not itself re-match (idempotent).
|
|
(
|
|
AUTHUSER_EMAIL_DOUBLE_ENCODED_PATTERN,
|
|
"authuser%3DSCRUBBED_EMAIL%40example.com",
|
|
),
|
|
# Unquoted-context fallback (mailto: hrefs, raw HTML/JS chunks).
|
|
(_EMAIL_PATTERN_BASE, "SCRUBBED_EMAIL@example.com"),
|
|
# -------------------------------------------------------------------------
|
|
# 6. Display names — JSON-key-anchored ONLY
|
|
# -------------------------------------------------------------------------
|
|
# We deliberately do NOT use a broad ``>[A-Z][a-z]+\s[A-Z][a-z]+<`` pattern
|
|
# here: that would clobber legitimate two-Capitalized-word fixture content
|
|
# such as ``>Source Title<`` in source-rename cassettes. Anchoring on the
|
|
# JSON key keeps the scrubber surgical.
|
|
(r"Google Account: [^\"<]+", "Google Account: SCRUBBED_NAME"),
|
|
(r'"displayName"\s*:\s*"[^"]+"', '"displayName":"SCRUBBED_NAME"'),
|
|
(r'"givenName"\s*:\s*"[^"]+"', '"givenName":"SCRUBBED_NAME"'),
|
|
(r'"familyName"\s*:\s*"[^"]+"', '"familyName":"SCRUBBED_NAME"'),
|
|
# Legacy hard-coded fixture name patterns kept for backward compatibility
|
|
# with cassettes recorded before the structural patterns above existed.
|
|
(r">People Conf<", ">SCRUBBED_NAME<"),
|
|
(r'"People Conf"', '"SCRUBBED_NAME"'),
|
|
# -------------------------------------------------------------------------
|
|
# 7. Playwright ``storage_state.json`` cookie objects
|
|
# -------------------------------------------------------------------------
|
|
# The header-form patterns above never fire on a serialized storage_state
|
|
# body, so we need explicit structural patterns for the JSON shape. The
|
|
# cookie-value match uses the escape-aware idiom ``[^"\\]*(?:\\.[^"\\]*)*``
|
|
# instead of the naive ``[^"]*``: the naive class terminates at the first
|
|
# ``"`` even when JSON-escaped (``\"``), which would silently leak the
|
|
# tail of a value containing a literal quote.
|
|
(
|
|
rf'("name":\s*"(?:{_COOKIE_JSON_NAMES_GROUP})"\s*,\s*"value":\s*")'
|
|
r'[^"\\]*(?:\\.[^"\\]*)*(")',
|
|
r"\1SCRUBBED\2",
|
|
),
|
|
(
|
|
r'("value":\s*")[^"\\]*(?:\\.[^"\\]*)*'
|
|
rf'("\s*,\s*"name":\s*"(?:{_COOKIE_JSON_NAMES_GROUP})")',
|
|
r"\1SCRUBBED\2",
|
|
),
|
|
# -------------------------------------------------------------------------
|
|
# 8. Direct JSON-dict-with-cookie-name-as-key shape: ``{"SID": "value"}``
|
|
# -------------------------------------------------------------------------
|
|
# ``is_clean`` detects this shape via ``_DETECT_COOKIE_JSON_KEY``; without a
|
|
# corresponding scrubber, a leak in this form would be unfixable by
|
|
# ``scrub_string`` (the validator would flag it but the sanitizer could
|
|
# never clean it). The value match uses the escape-aware idiom to match
|
|
# the other JSON-shape patterns above.
|
|
(
|
|
rf'("(?:{_COOKIE_JSON_NAMES_GROUP})"\s*:\s*")[^"\\]*(?:\\.[^"\\]*)*(")',
|
|
r"\1SCRUBBED\2",
|
|
),
|
|
# -------------------------------------------------------------------------
|
|
# 9. Upload tokens
|
|
# -------------------------------------------------------------------------
|
|
# X-GUploader-UploadID response header line. The token is a long random
|
|
# string that uniquely identifies a resumable-upload session.
|
|
(
|
|
r"X-GUploader-UploadID: [A-Za-z0-9_\-]+",
|
|
"X-GUploader-UploadID: SCRUBBED_UPLOAD_ID",
|
|
),
|
|
# Full upload URL that embeds the upload_id token in its query string.
|
|
# Match the whole URL (up to the next quote or whitespace) and collapse
|
|
# to a stable canonical form on NotebookLM's trusted upload endpoint so
|
|
# cassette replay still passes runtime upload-URL validation.
|
|
(
|
|
r"https://notebooklm\.google\.com/upload/_/\?[^\"\s]*upload_id=[A-Za-z0-9_\-]+",
|
|
"https://notebooklm.google.com/upload/_/?upload_id=SCRUBBED_UPLOAD_ID",
|
|
),
|
|
# Standalone upload_id query parameter (anywhere it appears outside the
|
|
# full upload URL above).
|
|
(r"upload_id=[A-Za-z0-9_\-]+", "upload_id=SCRUBBED_UPLOAD_ID"),
|
|
# -------------------------------------------------------------------------
|
|
# 10. Drive AONS tokens
|
|
# -------------------------------------------------------------------------
|
|
# AONS-prefixed strings are Drive permission/ACL tokens. The 20-char tail
|
|
# threshold avoids matching short literal "AONS" mentions in code or
|
|
# documentation while catching real tokens (which are typically 50+ chars).
|
|
(r"AONS[A-Za-z0-9_\-]{20,}", "SCRUBBED_AONS"),
|
|
# -------------------------------------------------------------------------
|
|
# 11. Drive file IDs — context-aware ONLY
|
|
# -------------------------------------------------------------------------
|
|
# Match ONLY inside Drive contexts: a ``"file_id": "..."`` JSON key or a
|
|
# ``/drive/v3/files/<id>`` URL path. Bare 33-44 char strings elsewhere
|
|
# are NOT scrubbed — that would false-positive on artifact IDs, source
|
|
# IDs, conversation IDs, and other internal NotebookLM identifiers.
|
|
(
|
|
r'("file_id"\s*:\s*")[A-Za-z0-9_\-]{33,44}(")',
|
|
r"\1SCRUBBED_DRIVE_FILE_ID\2",
|
|
),
|
|
(
|
|
r"(/drive/v3/files/)[A-Za-z0-9_\-]{33,44}",
|
|
r"\1SCRUBBED_DRIVE_FILE_ID",
|
|
),
|
|
# -------------------------------------------------------------------------
|
|
# 12. Escaped JSON display-name literals
|
|
# -------------------------------------------------------------------------
|
|
# Owner display names surface inside Google's sharing RPCs as positional
|
|
# list elements inside a stringified WRB payload, e.g.
|
|
# ``[\"alice@gmail.com\",1,[],[\"First Last\",\"https://lh3...\"]]``.
|
|
# The structured ``"displayName": "..."`` scrubbers in section 6 do not
|
|
# fire on the double-encoded form, so we add an escape-anchored pattern
|
|
# here. A negative lookahead carries the false-positive allowlist (font
|
|
# families, UI titles, artifact / notebook names) so that legitimate
|
|
# two-Capitalized-word fixture content is preserved — without that
|
|
# allowlist a broad ``\\"[A-Z][a-z]+(?: [A-Z][a-z]+)+\\"`` regex would
|
|
# clobber strings like ``\"Source Title\"`` during replay.
|
|
#
|
|
# The pattern requires the LITERAL escaped quotes (``\\"`` in regex,
|
|
# which is ``\"`` in the cassette body — a backslash followed by a real
|
|
# quote inside a JSON string) so it never fires on bare ``"Foo Bar"``
|
|
# JSON values; those are handled by the existing displayName /
|
|
# givenName / familyName JSON-key-anchored scrubbers in section 6.
|
|
(
|
|
rf'\\"(?!(?:{_DISPLAY_NAME_ALLOWLIST_ALT})\\")'
|
|
r'[A-Z][a-z]+(?: [A-Z][a-z]+)+\\"',
|
|
r'\\"SCRUBBED_NAME\\"',
|
|
),
|
|
# -------------------------------------------------------------------------
|
|
# 13. lh3.googleusercontent.com avatar URLs (both /a/ and /ogw/ paths)
|
|
# -------------------------------------------------------------------------
|
|
# Both the ``/a/`` and ``/ogw/`` path forms embed per-user avatar
|
|
# tokens. The character class includes ``=`` and ``-`` because the URL
|
|
# tail carries sizing modifiers (e.g. ``=s512``, ``=s32-c-mo``). The
|
|
# whole URL (scheme through token suffix) collapses to a single
|
|
# placeholder so that no fragment of the original token survives.
|
|
(
|
|
r"https?://lh3\.googleusercontent\.com/(?:a|ogw)/[A-Za-z0-9_=\-]+",
|
|
"SCRUBBED_AVATAR_URL",
|
|
),
|
|
]
|
|
|
|
|
|
# =============================================================================
|
|
# Public entry points
|
|
# =============================================================================
|
|
|
|
|
|
def scrub_string(text: str) -> str:
|
|
"""Apply every sensitive-pattern replacement to ``text``.
|
|
|
|
This is the single sanitization entry point consumed by
|
|
:mod:`tests.vcr_config` (and by future cassette tooling). The function is
|
|
idempotent on already-scrubbed content: each replacement embeds a sentinel
|
|
that does not itself match any pattern in :data:`SENSITIVE_PATTERNS`.
|
|
"""
|
|
for pattern, replacement in SENSITIVE_PATTERNS:
|
|
text = re.sub(pattern, replacement, text)
|
|
return text
|
|
|
|
|
|
# Pre-compiled detection-only patterns for :func:`is_clean`.
|
|
#
|
|
# ``is_clean`` is a *validator* — it must NOT modify text. It pulls cookie
|
|
# values out of every shape we know about and asks: "is this value one of the
|
|
# expected SCRUB_PLACEHOLDERS?" If not, it's a leak. The detection regexes
|
|
# differ from the scrub regexes in that they only need to extract the value;
|
|
# we lean on the placeholder allowlist to decide leak-or-not.
|
|
|
|
_COOKIE_NAMES_GROUP = (
|
|
"|".join(re.escape(name) for name in SESSION_COOKIES) + r"|__Secure-[^=\"]+|__Host-[^=\"]+"
|
|
)
|
|
|
|
_DETECT_COOKIE_HEADER = re.compile(
|
|
rf"(?<![A-Za-z0-9_-])(?P<name>{_COOKIE_NAMES_GROUP})=(?P<value>[^;\s]+)"
|
|
)
|
|
|
|
# Anchor used by :func:`find_cookie_header_leaks` to identify a cookie-header
|
|
# RUN inside arbitrary text. A cookie header is a ``;``-delimited run of
|
|
# ``name=value`` pairs; we recognize one as a *cookie header* (rather than an
|
|
# incidental ``k=v; k=v`` body fragment) when it contains at least one KNOWN
|
|
# session-cookie pair. That keeps the name-agnostic value check from firing on
|
|
# unrelated ``k=v`` content in response bodies while still catching every
|
|
# off-allowlist cookie (``_ga`` / ``_gcl_au`` / ``AEC`` …) riding in the same
|
|
# run. The recorder's :func:`scrub_request` / gate's YAML-aware pass operate on
|
|
# the isolated logical value via :func:`find_cookie_leaks`; this string-level
|
|
# helper is the in-text complement for :func:`is_clean`.
|
|
_COOKIE_RUN_ANCHOR = re.compile(rf"(?<![A-Za-z0-9_-])(?:{_COOKIE_NAMES_GROUP})=")
|
|
|
|
# Matches one cookie run on a SINGLE LINE: a chain of ``name=value`` pairs
|
|
# joined by ``;`` (with optional spaces). Crucially the value class is
|
|
# ``[^;\n]*`` — it must NOT cross a newline, or a greedy match would swallow the
|
|
# entire YAML body (every ``k: v`` header line that happens to follow) and the
|
|
# ``;``-split would then mis-read non-cookie header content (``ma=2592000``,
|
|
# ``charset=utf-8``, ``Priority=HIGH`` …) as leaking cookies. Cookie runs are
|
|
# line-local: a folded YAML cookie scalar that wraps a run across a physical
|
|
# newline is still caught by the gate's authoritative YAML-aware pass
|
|
# (:func:`tests.scripts.check_cassettes_clean._scan_cookie_headers_yaml`), which
|
|
# stitches the logical value back together before calling :func:`find_cookie_leaks`.
|
|
_COOKIE_RUN_RE = re.compile(r"[^=;\s]+=[^;\n]*(?:;[ \t]*[^=;\s]+=[^;\n]*)*")
|
|
|
|
|
|
def find_cookie_header_leaks(text: str) -> list[str]:
|
|
"""Name-agnostic cookie-value leak scan over ARBITRARY text (for is_clean).
|
|
|
|
Locates every single-line ``;``-delimited cookie RUN that contains at least
|
|
one known session-cookie pair (the :data:`_COOKIE_RUN_ANCHOR` discriminator
|
|
that keeps this from firing on incidental ``k=v; k=v`` body content), then
|
|
checks EVERY pair in that run — name-agnostically — via
|
|
:func:`find_cookie_leaks`. This is what makes :func:`is_clean` catch the
|
|
``_ga`` class: a Google-Analytics cookie sharing a line with ``SID=`` /
|
|
``__Secure-1PSID=`` is flagged even though its name is on no allowlist.
|
|
|
|
The run is line-local (the value class does not cross ``\\n``) so feeding
|
|
this whole-file text never over-matches across YAML header lines; a cookie
|
|
run a folded scalar split across physical lines is recovered by the gate's
|
|
YAML-aware pass instead. Returns human-readable
|
|
``"Leak (Cookie header): ..."`` strings.
|
|
"""
|
|
leaks: list[str] = []
|
|
seen: set[str] = set()
|
|
# Scan line-by-line and run the (backtracking-prone) cookie-run regex ONLY on
|
|
# physical lines that already carry a known session cookie. The cheap,
|
|
# non-backtracking ``_COOKIE_RUN_ANCHOR`` pre-filter keeps the expensive
|
|
# pair-extraction off large response-body lines — without it, ``finditer``
|
|
# over a multi-MB cassette body backtracks catastrophically (37s -> 0.05s on
|
|
# the 1.8MB flashcards cassette; identical results). Runs are line-local, so
|
|
# restricting to anchored lines finds exactly the same cookie pairs.
|
|
for line in text.splitlines():
|
|
if not _COOKIE_RUN_ANCHOR.search(line):
|
|
continue
|
|
for run_match in _COOKIE_RUN_RE.finditer(line):
|
|
run = run_match.group(0)
|
|
# Only treat this run as a cookie header if it carries a known session
|
|
# cookie — otherwise it's incidental ``k=v`` content, not a cookie line.
|
|
if not _COOKIE_RUN_ANCHOR.search(run):
|
|
continue
|
|
if run in seen:
|
|
continue
|
|
seen.add(run)
|
|
leaks.extend(find_cookie_leaks(run))
|
|
return leaks
|
|
|
|
|
|
_DETECT_COOKIE_JSON_NAME_FIRST = re.compile(
|
|
rf'"name"\s*:\s*"(?P<name>{_COOKIE_NAMES_GROUP})"\s*,\s*"value"\s*:\s*"'
|
|
r'(?P<value>(?:[^"\\]|\\.)*)"'
|
|
)
|
|
_DETECT_COOKIE_JSON_VALUE_FIRST = re.compile(
|
|
r'"value"\s*:\s*"(?P<value>(?:[^"\\]|\\.)*)"\s*,\s*"name"\s*:\s*"'
|
|
rf'(?P<name>{_COOKIE_NAMES_GROUP})"'
|
|
)
|
|
_DETECT_COOKIE_JSON_KEY = re.compile(
|
|
rf'"(?P<name>{_COOKIE_NAMES_GROUP})"\s*:\s*"(?P<value>(?:[^"\\]|\\.)*)"'
|
|
)
|
|
|
|
# WIZ_global_data and form-body token fields, in the same order as the
|
|
# corresponding scrubbers in ``SENSITIVE_PATTERNS``. Compiled at import time so
|
|
# repeated ``is_clean`` calls (one per cassette under CI) don't pay the cost.
|
|
# The value capture uses the same escape-aware idiom as the cookie-shape
|
|
# detectors above so a token containing a JSON-escaped quote (``\"``) is
|
|
# captured in full instead of truncated at the first literal quote.
|
|
_DETECT_TOKEN_FIELDS: list[tuple[str, re.Pattern[str]]] = [
|
|
("SNlM0e (CSRF)", re.compile(r'"SNlM0e"\s*:\s*"((?:[^"\\]|\\.)*)"')),
|
|
("FdrFJe (session)", re.compile(r'"FdrFJe"\s*:\s*"((?:[^"\\]|\\.)*)"')),
|
|
("oPEP7c (email)", re.compile(r'"oPEP7c"\s*:\s*"((?:[^"\\]|\\.)*)"')),
|
|
("S06Grb (user_id)", re.compile(r'"S06Grb"\s*:\s*"((?:[^"\\]|\\.)*)"')),
|
|
("W3Yyqf (user_id)", re.compile(r'"W3Yyqf"\s*:\s*"((?:[^"\\]|\\.)*)"')),
|
|
("qDCSke (user_id)", re.compile(r'"qDCSke"\s*:\s*"((?:[^"\\]|\\.)*)"')),
|
|
("B8SWKb (api_key)", re.compile(r'"B8SWKb"\s*:\s*"((?:[^"\\]|\\.)*)"')),
|
|
("VqImj (api_key)", re.compile(r'"VqImj"\s*:\s*"((?:[^"\\]|\\.)*)"')),
|
|
("JrWMbf (api_key)", re.compile(r'"JrWMbf"\s*:\s*"((?:[^"\\]|\\.)*)"')),
|
|
("QGcrse (client_id)", re.compile(r'"QGcrse"\s*:\s*"((?:[^"\\]|\\.)*)"')),
|
|
("iQJtYd (project_id)", re.compile(r'"iQJtYd"\s*:\s*"((?:[^"\\]|\\.)*)"')),
|
|
]
|
|
|
|
# Compiled detection-only pattern for emails (no replacement string baked in).
|
|
# Three-shape detector — mirrors the scrubber patterns in section 5 above:
|
|
# 1. Literal ``@`` form on the provider allowlist (JSON, mailto: hrefs).
|
|
# 2. URL-encoded ``authuser=<email>`` query-param form for *any* domain.
|
|
# 3. Double-encoded ``authuser%3D<email>`` redirect-param form (issue #1368).
|
|
_DETECT_EMAIL = re.compile(
|
|
r"[A-Za-z0-9._%+\-]+@(?:"
|
|
+ "|".join(EMAIL_PROVIDERS)
|
|
+ r")\.com"
|
|
+ r"|"
|
|
+ _AUTHUSER_EMAIL_PATTERN
|
|
+ r"|"
|
|
+ AUTHUSER_EMAIL_DOUBLE_ENCODED_PATTERN
|
|
)
|
|
|
|
# upload + Drive token detectors.
|
|
#
|
|
# Each entry is (label, regex) where the regex's group(1) captures the value
|
|
# that must match a known scrub placeholder. The regexes deliberately accept
|
|
# both raw tokens AND their canonical placeholder form so that an already-
|
|
# scrubbed cassette passes ``is_clean`` cleanly (idempotent validation).
|
|
_DETECT_UPLOAD_DRIVE_FIELDS: list[tuple[str, re.Pattern[str]]] = [
|
|
# X-GUploader-UploadID response header: value must be SCRUBBED_UPLOAD_ID.
|
|
("upload header", re.compile(r"X-GUploader-UploadID:\s*([A-Za-z0-9_\-]+)")),
|
|
# Standalone upload_id query parameter: value must be SCRUBBED_UPLOAD_ID.
|
|
# This intentionally matches BOTH dirty tokens and the canonical
|
|
# placeholder; the placeholder check below decides leak-or-clean.
|
|
("upload_id param", re.compile(r"upload_id=([A-Za-z0-9_\-]+)")),
|
|
# Drive AONS tokens — 20-char tail threshold avoids matching short
|
|
# literal ``AONS`` mentions in code or docs. The captured group is the
|
|
# FULL token (including the ``AONS`` prefix) so the placeholder
|
|
# ``SCRUBBED_AONS`` is matched directly.
|
|
("Drive AONS token", re.compile(r"(AONS[A-Za-z0-9_\-]{20,}|SCRUBBED_AONS)")),
|
|
# Drive file ID in JSON key: ``"file_id": "<id>"``.
|
|
("Drive file_id (JSON)", re.compile(r'"file_id"\s*:\s*"([^"]+)"')),
|
|
# Drive file ID in URL: ``/drive/v3/files/<id>``.
|
|
("Drive file_id (URL)", re.compile(r"/drive/v3/files/([A-Za-z0-9_\-]+)")),
|
|
]
|
|
|
|
# Full upload URL is rewritten to a trusted-host placeholder with a scrubbed
|
|
# upload_id. Any NotebookLM upload URL carrying a non-placeholder upload_id is
|
|
# a leak.
|
|
_DETECT_UPLOAD_URL = re.compile(
|
|
r"https://notebooklm\.google\.com/upload/_/\?[^\"\s]*upload_id=(?!SCRUBBED_UPLOAD_ID\b)"
|
|
)
|
|
|
|
# escaped JSON display-name literal detector.
|
|
#
|
|
# Matches ``\"First Last\"`` inside a double-encoded JSON string. The
|
|
# false-positive allowlist is consulted at the call site in ``is_clean``
|
|
# rather than baked into the regex so the detector stays simple and the
|
|
# allowlist remains observable. The captured group is the inner name (no
|
|
# escape quotes) so we can compare it to DISPLAY_NAME_FALSE_POSITIVES
|
|
# directly.
|
|
_DETECT_DISPLAY_NAME_ESCAPED = re.compile(r'\\"([A-Z][a-z]+(?: [A-Z][a-z]+)+)\\"')
|
|
|
|
# avatar URL detector (/ogw/ group). The pattern matches
|
|
# both ``/a/`` and ``/ogw/`` path forms. The scrubber collapses the entire
|
|
# URL to ``SCRUBBED_AVATAR_URL``, so any match here is by definition a
|
|
# leak (the placeholder string doesn't itself contain ``lh3.``).
|
|
_DETECT_AVATAR_URL = re.compile(r"https?://lh3\.googleusercontent\.com/(?:a|ogw)/[A-Za-z0-9_=\-]+")
|
|
|
|
# Catch-all auth-token detector — the validator twin of
|
|
# :data:`_AUTH_TOKEN_PATTERNS`. The scrubber collapses every ``g.a000-...`` /
|
|
# ``sidts-...`` / ``ya29....`` token to ``SCRUBBED`` (which contains none of
|
|
# these prefixes), so ANY match here is by definition an unredacted leak —
|
|
# regardless of which cookie name or body field carried it. This is the guard
|
|
# rail that would have caught the ``LSID`` leak: it never depended on ``LSID``
|
|
# being on the cookie allowlist.
|
|
_DETECT_AUTH_TOKEN = re.compile("|".join(_AUTH_TOKEN_PATTERNS))
|
|
|
|
# Google API-key shape detector — the validator twin of
|
|
# :data:`_GOOGLE_API_KEY_PATTERN`. The scrubber collapses every ``AIza...`` key
|
|
# to the ``SCRUBBED_API_KEY`` sentinel (which does not contain the ``AIza``
|
|
# prefix), so ANY match here is by definition an unredacted leak regardless of
|
|
# which WIZ field carried it. This is the field-name-agnostic backstop that
|
|
# closes the ``JrWMbf`` gap.
|
|
_DETECT_GOOGLE_API_KEY = re.compile(_GOOGLE_API_KEY_PATTERN)
|
|
|
|
# Double-encoded ``authuser%3D<email>`` shape detector (issue #1368). Unlike the
|
|
# single-encoded ``authuser=<email>`` form — which only ``is_clean`` runs,
|
|
# because the literal-``@`` provider branch of ``_DETECT_EMAIL`` also fires on
|
|
# legitimate placeholder content like ``alice@gmail.com`` — the double-encoded
|
|
# ``authuser%3D…%40…`` shape has NO legitimate occurrence anywhere in the repo,
|
|
# so it is safe to run over arbitrary fixture directories via
|
|
# :data:`_CREDENTIAL_DETECTORS`. The negative lookahead excludes the canonical
|
|
# ``authuser%3DSCRUBBED_EMAIL%40example.com`` placeholder so an already-scrubbed
|
|
# cassette validates cleanly (idempotent).
|
|
_DETECT_AUTHUSER_EMAIL_DOUBLE_ENCODED = re.compile(
|
|
r"authuser%3D(?!SCRUBBED_EMAIL%40example\.com)" + _AUTHUSER_EMAIL_TAIL
|
|
)
|
|
|
|
|
|
# Detectors with ZERO legitimate-occurrence risk anywhere in the repository:
|
|
# raw Google auth-token shapes (``g.a000-`` / ``sidts-`` / ``ya29.``), the
|
|
# canonical Google API-key shape (``AIza`` + 35 chars), and the double-encoded
|
|
# ``authuser%3D<email>`` redirect-param shape (issue #1368). Unlike the
|
|
# cookie-value / display-name / email heuristics that :func:`is_clean` also runs
|
|
# — which intentionally fire on placeholder fixture content such as ``"Scrubbed
|
|
# Note Title"`` or ``alice@gmail.com`` — these shapes never match legitimate
|
|
# test data, so they are safe to run over arbitrary files (golden JSON/HTML
|
|
# fixtures, docs, source) with no per-file allowlist. This is what the
|
|
# ``--secrets-only`` mode of ``check_cassettes_clean.py`` uses to extend leak
|
|
# detection beyond ``tests/cassettes/`` without drowning in false positives.
|
|
_CREDENTIAL_DETECTORS: list[tuple[str, re.Pattern[str]]] = [
|
|
("auth token", _DETECT_AUTH_TOKEN),
|
|
("Google API key", _DETECT_GOOGLE_API_KEY),
|
|
("double-encoded authuser email", _DETECT_AUTHUSER_EMAIL_DOUBLE_ENCODED),
|
|
]
|
|
|
|
|
|
# =============================================================================
|
|
# Generic high-entropy / unknown-field credential scan
|
|
# =============================================================================
|
|
#
|
|
# THE KNOWN-SHAPE BOUNDARY (residual-risk decision; ADR-0006, issue #1382).
|
|
# ---------------------------------------------------------------------------
|
|
# Everything ABOVE this point is *name-anchored* (cookie names, WIZ field IDs)
|
|
# or *known-shape* (``g.a000-`` / ``sidts-`` / ``ya29.`` / ``AIza`` prefixes).
|
|
# That makes the guard NECESSARY-but-not-SUFFICIENT: a credential family the
|
|
# registry does not yet know about — a NOVEL token prefix, or a known secret
|
|
# riding in an un-targeted JSON field — passes the targeted detectors silently.
|
|
# This is not hypothetical: the guard has missed two such shapes historically
|
|
# (the ``LSID`` cookie whose name was off the allowlist, and the ``JrWMbf`` WIZ
|
|
# field whose API key the name-anchored scrubbers skipped), and #1372 fixed a
|
|
# double-encoded-email shape that leaked the maintainer's address. Each fix
|
|
# extended the known-shape set by ONE entry — reactive, after-the-leak.
|
|
#
|
|
# This scanner is the deliberate, field-agnostic complement: it flags a quoted
|
|
# JSON string scalar whose ENTIRE value is one long, high-entropy
|
|
# base64url/hex run, regardless of the field name carrying it. It is the
|
|
# "would have caught it generically" backstop the targeted detectors lack.
|
|
#
|
|
# Why it is SCOPED to a *quoted scalar value* rather than scanning every token:
|
|
# committed cassettes are dense with legitimate high-entropy text — minified
|
|
# CSS class names, ``fonts.gstatic.com`` / ``googleusercontent.com`` URLs,
|
|
# 1.5 KB hex mind-map blobs, and 600-char ``context=eJw…`` zlib-base64 query
|
|
# params. A raw per-token entropy scan would drown in thousands of those false
|
|
# positives. But NONE of that legitimate content appears as a *clean quoted
|
|
# JSON scalar* (``"field":"<one-pure-token>"``) — which is exactly the shape a
|
|
# leaked credential takes when it rides in an unknown field. Anchoring on that
|
|
# shape is what makes ZERO false positives across every committed cassette
|
|
# achievable while still catching a planted novel token. This residual-risk
|
|
# property is therefore a DECISION, not an accident: anything that is NOT a
|
|
# quoted high-entropy scalar (a credential split across structural punctuation,
|
|
# a token in a non-JSON encoding) remains the backstop's blind spot, covered by
|
|
# GitHub secret-scanning and the name-anchored detectors above.
|
|
#
|
|
# Thresholds (calibrated against every committed cassette — see the unit tests
|
|
# ``test_entropy_scan_*`` and ``test_all_committed_cassettes_pass_entropy_scan``):
|
|
#
|
|
# * base64url / mixed-alphabet tokens: length >= 40 AND Shannon entropy
|
|
# >= 4.0 bits/char. A 40+ char run drawn from a ~64-symbol alphabet with
|
|
# >=4.0 entropy is a random secret by construction; legitimate quoted
|
|
# scalars in cassettes (UUIDs, opaque IDs) never reach both bars at once.
|
|
# * pure-hex tokens (``[0-9a-fA-F]`` only): a separate length floor of 64,
|
|
# because a 16-symbol alphabet CAPS Shannon entropy at log2(16) = 4.0
|
|
# bits/char — a hex secret can never reliably clear the 4.0 bar, so length
|
|
# alone gates it. 64 hex chars = 256 bits, comfortably above any incidental
|
|
# hex literal (a 32-hex MD5 or a 36-char UUID stays well under the floor).
|
|
#
|
|
# Allowlist (known-benign long tokens that must NOT be flagged):
|
|
#
|
|
# * the canonical ``SCRUB_PLACEHOLDERS`` sentinels (idempotent validation);
|
|
# * the canonical UUID shape (``8-4-4-4-12`` hex-with-dashes) — these are
|
|
# NotebookLM-internal artifact / source / conversation IDs that the
|
|
# surrounding scrubbers deliberately preserve. The base64 length+entropy
|
|
# gate already excludes them (36 chars < 40, dashes depress entropy), but
|
|
# the explicit shape skip documents the intent and guards against a future
|
|
# threshold change re-flagging them.
|
|
_ENTROPY_MIN_LEN_BASE64 = 40
|
|
_ENTROPY_MIN_BITS = 4.0
|
|
_ENTROPY_MIN_LEN_HEX = 64
|
|
|
|
# Cap the substring fed to the entropy computation. A cassette can carry a
|
|
# multi-megabyte base64-encoded asset as one quoted scalar (an inline image /
|
|
# PDF / audio blob), and computing Shannon entropy over the whole run would be a
|
|
# needless O(n) hot loop per scanned line. A random credential is high-entropy
|
|
# throughout, so a 512-char prefix is a faithful representative — the entropy of
|
|
# the prefix and the whole token converge well before this bound (gemini review
|
|
# on #1387). Length gating still uses the FULL token length, not the cap.
|
|
_ENTROPY_SAMPLE_CHARS = 512
|
|
|
|
# A quoted JSON string scalar whose entire content is a single base64url / hex
|
|
# token. ``\A``/``\Z`` inside the captured group are unnecessary because the
|
|
# surrounding quotes already anchor the run to the FULL value — a token split
|
|
# by any non-``[A-Za-z0-9_\-+/=]`` byte simply won't match.
|
|
_DETECT_QUOTED_TOKEN = re.compile(r'"([A-Za-z0-9_\-+/=]+)"')
|
|
|
|
# Canonical UUID shape (``8-4-4-4-12``). Internal NotebookLM IDs, never secrets.
|
|
_UUID_SHAPE = re.compile(
|
|
r"\A[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}\Z"
|
|
)
|
|
|
|
# Pure-hex run (no base64-only symbols). Used to route a token onto the hex
|
|
# length floor instead of the entropy bar (hex entropy caps at 4.0).
|
|
_HEX_ONLY = re.compile(r"\A[0-9a-fA-F]+\Z")
|
|
|
|
|
|
def _shannon_entropy(token: str) -> float:
|
|
"""Shannon entropy of ``token`` in bits per character.
|
|
|
|
A measure of per-character randomness: a token drawn uniformly from an
|
|
``N``-symbol alphabet approaches ``log2(N)`` bits/char, while repetitive or
|
|
small-alphabet content scores lower. Returns ``0.0`` for the empty string.
|
|
|
|
Only the first :data:`_ENTROPY_SAMPLE_CHARS` characters are analyzed so a
|
|
multi-megabyte base64 asset scalar does not turn this into a per-line hot
|
|
loop; a random credential is high-entropy throughout, so the prefix is
|
|
representative.
|
|
"""
|
|
if not token:
|
|
return 0.0
|
|
sample = token[:_ENTROPY_SAMPLE_CHARS]
|
|
length = len(sample)
|
|
return -sum((count / length) * math.log2(count / length) for count in Counter(sample).values())
|
|
|
|
|
|
def _is_high_entropy_token(token: str) -> bool:
|
|
"""Return ``True`` if ``token`` looks like a novel high-entropy credential.
|
|
|
|
See the KNOWN-SHAPE BOUNDARY block above for the full rationale and the
|
|
threshold calibration. In brief: a pure base64url/hex run that is long
|
|
AND random enough that no legitimate quoted-scalar cassette value reaches
|
|
both bars, with the placeholder + UUID allowlist applied first.
|
|
"""
|
|
if token in SCRUB_PLACEHOLDERS:
|
|
return False
|
|
if _UUID_SHAPE.match(token):
|
|
return False
|
|
if _HEX_ONLY.match(token):
|
|
# Hex entropy caps at log2(16) == 4.0 bits/char, so length alone gates
|
|
# a hex secret; the entropy bar below would never reliably fire on it.
|
|
return len(token) >= _ENTROPY_MIN_LEN_HEX
|
|
if len(token) < _ENTROPY_MIN_LEN_BASE64:
|
|
return False
|
|
return _shannon_entropy(token) >= _ENTROPY_MIN_BITS
|
|
|
|
|
|
def find_high_entropy_leaks(text: str) -> list[str]:
|
|
"""Flag quoted high-entropy base64/hex scalars in ANY field (issue #1382).
|
|
|
|
The field-agnostic complement to the name-anchored / known-shape detectors:
|
|
catches a novel credential shape — or a known secret riding in an
|
|
un-targeted JSON field — that the targeted scrubbers miss. Scoped to a
|
|
*quoted JSON string scalar* so it never fires on the legitimate high-entropy
|
|
text (CSS, URLs, compressed-query blobs) that pervades real cassettes. See
|
|
the KNOWN-SHAPE BOUNDARY block above for the boundary this deliberately
|
|
draws. Each returned string is a human-readable ``"Leak (...): '...'"``.
|
|
"""
|
|
leaks: list[str] = []
|
|
for match in _DETECT_QUOTED_TOKEN.finditer(text):
|
|
token = match.group(1)
|
|
if _is_high_entropy_token(token):
|
|
leaks.append(f"Leak (high-entropy token): {token!r}")
|
|
return leaks
|
|
|
|
|
|
def find_credential_leaks(text: str) -> list[str]:
|
|
"""Return high-severity credential leaks (auth tokens + Google API keys).
|
|
|
|
A focused subset of :func:`is_clean` that runs ONLY the credential-shape
|
|
detectors in :data:`_CREDENTIAL_DETECTORS` PLUS the field-agnostic
|
|
high-entropy scan (:func:`find_high_entropy_leaks`). These shapes have no
|
|
legitimate occurrence anywhere in the repo, so unlike :func:`is_clean` this
|
|
is safe to point at fixture directories full of intentional placeholder
|
|
content. Each returned string is a human-readable ``"Leak (...): '...'"``
|
|
description.
|
|
"""
|
|
leaks: list[str] = []
|
|
for label, regex in _CREDENTIAL_DETECTORS:
|
|
for match in regex.finditer(text):
|
|
leaks.append(f"Leak ({label}): {match.group(0)!r}")
|
|
leaks.extend(find_high_entropy_leaks(text))
|
|
return leaks
|
|
|
|
|
|
def is_clean(text: str) -> tuple[bool, list[str]]:
|
|
"""Validate that ``text`` contains no unredacted sensitive data.
|
|
|
|
Closes the cookie-value leak heuristic: cleanliness is judged by exact
|
|
membership in :data:`SCRUB_PLACEHOLDERS`, NOT by the legacy "starts with
|
|
S" character-class heuristic that allowed any real secret beginning with
|
|
``S`` (and there are plenty — SID values, SAPISID values, OAuth ``state``
|
|
tokens) to slip past the guard.
|
|
|
|
Parameters
|
|
----------
|
|
text:
|
|
The full text of a cassette (or any string) to inspect.
|
|
|
|
Returns
|
|
-------
|
|
``(ok, leaks)`` where ``ok`` is ``True`` iff ``leaks`` is empty. Each leak
|
|
string is a human-readable description suitable for printing in CI output.
|
|
|
|
Display-name + avatar coverage
|
|
---------------------------------------
|
|
Escaped display-name literals (``\\"First Last\\"`` inside double-
|
|
encoded WRB payloads) and ``lh3.googleusercontent.com/(a|ogw)/`` avatar
|
|
URLs are BOTH scrubbed and detected. The display-name detector consults
|
|
:data:`DISPLAY_NAME_FALSE_POSITIVES` so legitimate two-Capitalized-word
|
|
fixture content (font families, UI titles, artifact / notebook names)
|
|
is not flagged. The structured ``"displayName": "..."``-style fields
|
|
from section 6 remain scrub-only (no detector) — that gap is harmless
|
|
because A6a's escape-anchored detector also catches the equivalent
|
|
leak shape inside any stringified WRB payload, which is where these
|
|
fields actually surface.
|
|
"""
|
|
leaks: list[str] = []
|
|
|
|
# --- 1. Cookie shapes ---------------------------------------------------
|
|
seen: set[tuple[str, str]] = set()
|
|
for regex, shape in (
|
|
(_DETECT_COOKIE_HEADER, "cookie header"),
|
|
(_DETECT_COOKIE_JSON_NAME_FIRST, "storage_state (name-first)"),
|
|
(_DETECT_COOKIE_JSON_VALUE_FIRST, "storage_state (value-first)"),
|
|
(_DETECT_COOKIE_JSON_KEY, "JSON key"),
|
|
):
|
|
for match in regex.finditer(text):
|
|
name = match.group("name")
|
|
value = match.group("value")
|
|
key = (name, value)
|
|
if key in seen:
|
|
continue
|
|
seen.add(key)
|
|
if value not in SCRUB_PLACEHOLDERS:
|
|
leaks.append(
|
|
f"Leak ({shape}): cookie {name!r} value {value!r} is not"
|
|
f" a known scrub placeholder"
|
|
)
|
|
|
|
# --- 1b. Name-AGNOSTIC cookie-value scan -------------------------------
|
|
# The name-anchored shapes above only flag cookies on the allowlist. This
|
|
# pass flags ANY cookie pair (``_ga`` / ``_gcl_au`` / ``AEC`` / an unknown
|
|
# ``foo``) whose value is not a placeholder, scoped to a ``;``-delimited run
|
|
# that carries a known session cookie so it never fires on incidental
|
|
# ``k=v`` body content. This is the check that catches the ``_ga`` class
|
|
# going forward (the recorder + gate clear it via the name-agnostic
|
|
# scrubber). See :func:`find_cookie_header_leaks`.
|
|
leaks.extend(find_cookie_header_leaks(text))
|
|
|
|
# --- 2. Real email addresses (any provider we redact) -------------------
|
|
# Skip canonical placeholders so the provider-agnostic ``authuser=``
|
|
# branch of ``_DETECT_EMAIL`` (which matches any TLD) doesn't re-flag
|
|
# the scrubbed replacement on a second pass.
|
|
for match in _DETECT_EMAIL.finditer(text):
|
|
matched = match.group(0)
|
|
if matched in SCRUB_PLACEHOLDERS:
|
|
continue
|
|
leaks.append(f"Leak (email): {matched!r}")
|
|
|
|
# --- 3. Token / ID fields that should be redacted ----------------------
|
|
for label, regex in _DETECT_TOKEN_FIELDS:
|
|
for match in regex.finditer(text):
|
|
value = match.group(1)
|
|
if value not in SCRUB_PLACEHOLDERS:
|
|
leaks.append(f"Leak ({label}): {value!r}")
|
|
|
|
# --- 4. Upload + Drive token fields ------------------------------------
|
|
for label, regex in _DETECT_UPLOAD_DRIVE_FIELDS:
|
|
for match in regex.finditer(text):
|
|
value = match.group(1)
|
|
if value not in SCRUB_PLACEHOLDERS:
|
|
leaks.append(f"Leak ({label}): {value!r}")
|
|
|
|
# --- 5. Full upload URL -----------------------------------------------
|
|
# The scrubber preserves the trusted host/path but rewrites upload_id to
|
|
# SCRUBBED_UPLOAD_ID, so any match here is by definition a leak.
|
|
for match in _DETECT_UPLOAD_URL.finditer(text):
|
|
leaks.append(f"Leak (upload URL): {match.group(0)!r}")
|
|
|
|
# --- 6. Escaped display-name literals ----------------------------------
|
|
# The false-positive allowlist (font families, UI titles, artifact /
|
|
# notebook names) is consulted here rather than baked into the regex so
|
|
# the detector stays simple and the allowlist remains observable.
|
|
for match in _DETECT_DISPLAY_NAME_ESCAPED.finditer(text):
|
|
inner = match.group(1)
|
|
if inner in DISPLAY_NAME_FALSE_POSITIVES:
|
|
continue
|
|
leaks.append(f"Leak (escaped display name): {match.group(0)!r}")
|
|
|
|
# --- 7. Avatar URLs ---------------------------------------------------
|
|
# The scrubber collapses the whole URL to ``SCRUBBED_AVATAR_URL``, so any
|
|
# match of the raw URL form here is by definition a leak.
|
|
for match in _DETECT_AVATAR_URL.finditer(text):
|
|
leaks.append(f"Leak (avatar URL): {match.group(0)!r}")
|
|
|
|
# --- 8. Catch-all Google auth-token shapes -----------------------------
|
|
# ``g.a000-...`` / ``sidts-...`` / ``ya29....`` tokens are scrubbed to
|
|
# ``SCRUBBED`` wherever they appear (cookie values on or off the allowlist,
|
|
# response bodies, headers). Any surviving raw token is a leak by
|
|
# definition — this is the cookie-name-agnostic backstop that closes the
|
|
# ``LSID`` gap.
|
|
for match in _DETECT_AUTH_TOKEN.finditer(text):
|
|
leaks.append(f"Leak (auth token): {match.group(0)!r}")
|
|
|
|
# --- 9. Google API-key shape (field-name-agnostic) ---------------------
|
|
# ``AIza...`` keys are scrubbed to ``SCRUBBED_API_KEY`` wherever they
|
|
# appear, regardless of the WIZ field name carrying them. Any surviving raw
|
|
# key is a leak by definition — this is the backstop that closes the
|
|
# ``JrWMbf`` gap the name-anchored field detectors missed.
|
|
for match in _DETECT_GOOGLE_API_KEY.finditer(text):
|
|
leaks.append(f"Leak (Google API key): {match.group(0)!r}")
|
|
|
|
# --- 10. Generic high-entropy / unknown-field credential scan ----------
|
|
# Field-agnostic backstop for a NOVEL credential shape — or a known secret
|
|
# in an un-targeted JSON field — that the name-anchored / known-shape
|
|
# detectors above miss (issue #1382). Scoped to a quoted high-entropy
|
|
# base64/hex scalar so it never fires on the legitimate high-entropy text
|
|
# (CSS, URLs, compressed-query blobs) that pervades real cassettes. See the
|
|
# KNOWN-SHAPE BOUNDARY block near :func:`find_high_entropy_leaks`.
|
|
leaks.extend(find_high_entropy_leaks(text))
|
|
|
|
return (not leaks, leaks)
|
|
|
|
|
|
# =============================================================================
|
|
# Synthetic error-response builders for VCR recording
|
|
# =============================================================================
|
|
#
|
|
# These helpers exist so error-shape cassettes can be generated whose
|
|
# responses match the shapes our client's exception mapping (see
|
|
# :mod:`notebooklm._runtime.helpers` for ``is_auth_error`` and the retry
|
|
# middleware for 429/5xx) keys on:
|
|
#
|
|
# - HTTP 429 -> ``TransportRateLimited`` -> ``RateLimitError``
|
|
# - HTTP 5xx -> ``TransportServerError`` -> ``ServerError``
|
|
# - HTTP 400 -> ``is_auth_error()`` -> refresh path + ``AuthError`` on
|
|
# second failure
|
|
#
|
|
# The synthetic bodies are **not** captured from Google. They are deliberately
|
|
# minimal and exist purely to validate client-side exception mapping. Documented
|
|
# warning lives in ``docs/development.md`` under "Synthetic error cassettes".
|
|
|
|
ERROR_MODE_RATE_LIMIT = "429"
|
|
ERROR_MODE_SERVER = "5xx"
|
|
ERROR_MODE_EXPIRED_CSRF = "expired_csrf"
|
|
|
|
VALID_ERROR_MODES: frozenset[str] = frozenset(
|
|
{ERROR_MODE_RATE_LIMIT, ERROR_MODE_SERVER, ERROR_MODE_EXPIRED_CSRF}
|
|
)
|
|
|
|
# Filename prefix that error-cassette generators MUST apply to cassettes
|
|
# produced through this plumbing. The prefix is mechanical: it lets a
|
|
# reader of ``tests/cassettes/`` distinguish synthetic error shapes from real
|
|
# recordings at a glance, without having to open the YAML.
|
|
SYNTHETIC_ERROR_CASSETTE_PREFIX = "error_synthetic_"
|
|
|
|
|
|
def synthetic_error_cassette_name(mode: str, slug: str) -> str:
|
|
"""Build the canonical ``error_synthetic_<mode>_<slug>.yaml`` filename.
|
|
|
|
Args:
|
|
mode: One of ``VALID_ERROR_MODES``.
|
|
slug: A short identifier for the RPC being recorded (e.g. ``"list_notebooks"``).
|
|
|
|
Raises:
|
|
ValueError: If ``mode`` is not a recognized synthetic-error mode.
|
|
"""
|
|
if mode not in VALID_ERROR_MODES:
|
|
raise ValueError(
|
|
f"Unknown synthetic error mode {mode!r}. Valid modes: {sorted(VALID_ERROR_MODES)}"
|
|
)
|
|
return f"{SYNTHETIC_ERROR_CASSETTE_PREFIX}{mode}_{slug}.yaml"
|
|
|
|
|
|
def build_synthetic_error_response(
|
|
mode: str,
|
|
) -> tuple[int, bytes, dict[str, str]]:
|
|
"""Return a ``(status_code, body, headers)`` triple for a synthetic error.
|
|
|
|
The shape is intentionally minimal; the client's exception mapping keys on
|
|
the HTTP status code (see :func:`notebooklm._runtime.helpers.is_auth_error`
|
|
and the 429 / 5xx branches in the retry middleware), so a
|
|
syntactically-valid Google error-shaped body is sufficient.
|
|
|
|
For the ``expired_csrf`` mode we return HTTP 400 — not 401 — because that
|
|
matches the documented Google contract: NotebookLM returns 400 (not 401/403)
|
|
when the embedded CSRF token has expired, which is why ``is_auth_error``
|
|
treats 400 as an auth-refresh trigger. See
|
|
:func:`notebooklm._runtime.helpers.is_auth_error`.
|
|
|
|
Args:
|
|
mode: One of ``VALID_ERROR_MODES``.
|
|
|
|
Returns:
|
|
A tuple of ``(status_code, body_bytes, headers_dict)`` suitable for
|
|
constructing an ``httpx.Response``.
|
|
|
|
Raises:
|
|
ValueError: If ``mode`` is not a recognized synthetic-error mode.
|
|
"""
|
|
if mode == ERROR_MODE_RATE_LIMIT:
|
|
body = (
|
|
b'{"error": {"code": 429, "message": "Rate limited", "status": "RESOURCE_EXHAUSTED"}}'
|
|
)
|
|
# Retry-After is honored by the 429 retry middleware in the shared
|
|
# authed transport.
|
|
# Setting a small value keeps the recording-time loop short.
|
|
headers = {
|
|
"Content-Type": "application/json; charset=UTF-8",
|
|
"Retry-After": "1",
|
|
}
|
|
return (429, body, headers)
|
|
if mode == ERROR_MODE_SERVER:
|
|
body = b'{"error": {"code": 500, "message": "Internal error"}}'
|
|
headers = {"Content-Type": "application/json; charset=UTF-8"}
|
|
return (500, body, headers)
|
|
if mode == ERROR_MODE_EXPIRED_CSRF:
|
|
# NotebookLM returns 400 (not 401/403) for expired CSRF — this matches
|
|
# the ``is_auth_error`` branch that treats 400/401/403 as auth-refresh
|
|
# triggers. The body shape echoes Google's typical "invalid request"
|
|
# response; the client keys on status code, not body, for this path.
|
|
body = (
|
|
b'{"error": {"code": 400, "message": "Invalid request token", '
|
|
b'"status": "INVALID_ARGUMENT"}}'
|
|
)
|
|
headers = {"Content-Type": "application/json; charset=UTF-8"}
|
|
return (400, body, headers)
|
|
raise ValueError(
|
|
f"Unknown synthetic error mode {mode!r}. Valid modes: {sorted(VALID_ERROR_MODES)}"
|
|
)
|