chore: import upstream snapshot with attribution
Test / Code Quality (push) Has been cancelled
Test / Test (macos-latest, Python 3.10) (push) Has been cancelled
Test / Test (macos-latest, Python 3.11) (push) Has been cancelled
Test / Test (macos-latest, Python 3.12) (push) Has been cancelled
Test / Test (macos-latest, Python 3.13) (push) Has been cancelled
Test / Test (macos-latest, Python 3.14) (push) Has been cancelled
Test / Test (ubuntu-latest, Python 3.10) (push) Has been cancelled
Test / Test (ubuntu-latest, Python 3.11) (push) Has been cancelled
Test / Test (ubuntu-latest, Python 3.12) (push) Has been cancelled
Test / Test (ubuntu-latest, Python 3.13) (push) Has been cancelled
Test / Test (ubuntu-latest, Python 3.14) (push) Has been cancelled
Test / Test (windows-latest, Python 3.10) (push) Has been cancelled
Test / Test (windows-latest, Python 3.11) (push) Has been cancelled
Test / Test (windows-latest, Python 3.12) (push) Has been cancelled
Test / Test (windows-latest, Python 3.13) (push) Has been cancelled
Test / Test (windows-latest, Python 3.14) (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
dependency-audit / pip-audit (push) Has been cancelled
Test / Code Quality (push) Has been cancelled
Test / Test (macos-latest, Python 3.10) (push) Has been cancelled
Test / Test (macos-latest, Python 3.11) (push) Has been cancelled
Test / Test (macos-latest, Python 3.12) (push) Has been cancelled
Test / Test (macos-latest, Python 3.13) (push) Has been cancelled
Test / Test (macos-latest, Python 3.14) (push) Has been cancelled
Test / Test (ubuntu-latest, Python 3.10) (push) Has been cancelled
Test / Test (ubuntu-latest, Python 3.11) (push) Has been cancelled
Test / Test (ubuntu-latest, Python 3.12) (push) Has been cancelled
Test / Test (ubuntu-latest, Python 3.13) (push) Has been cancelled
Test / Test (ubuntu-latest, Python 3.14) (push) Has been cancelled
Test / Test (windows-latest, Python 3.10) (push) Has been cancelled
Test / Test (windows-latest, Python 3.11) (push) Has been cancelled
Test / Test (windows-latest, Python 3.12) (push) Has been cancelled
Test / Test (windows-latest, Python 3.13) (push) Has been cancelled
Test / Test (windows-latest, Python 3.14) (push) Has been cancelled
CodeQL / Analyze (push) Has been cancelled
dependency-audit / pip-audit (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,21 @@
|
||||
# Build-context excludes for deploy/Dockerfile (context = repo root).
|
||||
# Keeps the context small and never bakes secrets/venvs/caches into the image.
|
||||
.git
|
||||
.venv
|
||||
.venv*/
|
||||
.worktrees/
|
||||
**/__pycache__/
|
||||
*.py[cod]
|
||||
.pytest_cache/
|
||||
.mypy_cache/
|
||||
.ruff_cache/
|
||||
.coverage
|
||||
htmlcov/
|
||||
dist/
|
||||
build/
|
||||
*.egg-info/
|
||||
.sisyphus/
|
||||
node_modules/
|
||||
# never copy deploy secrets into the build
|
||||
deploy/profile/
|
||||
deploy/.env
|
||||
@@ -0,0 +1,35 @@
|
||||
# NotebookLM Configuration
|
||||
# Copy this file to .env and fill in your values
|
||||
|
||||
# =============================================================================
|
||||
# Configuration Environment Variables
|
||||
# =============================================================================
|
||||
|
||||
# Custom home directory for all config files (optional)
|
||||
# Default: ~/.notebooklm
|
||||
# NOTEBOOKLM_HOME=/custom/path
|
||||
|
||||
# Inline authentication JSON for CI/CD (optional)
|
||||
# When set, authentication is read from this variable instead of a file
|
||||
# Get the value from: cat $NOTEBOOKLM_HOME/storage_state.json (default: ~/.notebooklm)
|
||||
# NOTEBOOKLM_AUTH_JSON='{"cookies":[...]}'
|
||||
|
||||
# Enable RPC debug logging (optional)
|
||||
# NOTEBOOKLM_DEBUG_RPC=1
|
||||
|
||||
# =============================================================================
|
||||
# E2E Testing Configuration
|
||||
# =============================================================================
|
||||
|
||||
# Required for E2E tests: Your READ-ONLY test notebook ID
|
||||
# Create a notebook at https://notebooklm.google.com with:
|
||||
# - Multiple sources (text, URL, PDF, etc.)
|
||||
# - Some pre-generated artifacts (audio, quiz, etc.)
|
||||
# Copy the notebook ID from the URL: notebooklm.google.com/notebook/YOUR_ID
|
||||
# This notebook is used for READ-ONLY tests (list, get, download operations)
|
||||
NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID=your-notebook-id-here
|
||||
|
||||
# Optional: Generation test notebook ID
|
||||
# If not set, a notebook will be auto-created and its ID stored in
|
||||
# NOTEBOOKLM_HOME/generation_notebook_id for reuse across test runs.
|
||||
# NOTEBOOKLM_GENERATION_NOTEBOOK_ID=your-generation-notebook-id
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
name: Bug Report
|
||||
about: Report a bug in notebooklm-py
|
||||
title: ""
|
||||
labels: bug
|
||||
assignees: ""
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
A clear description of the bug.
|
||||
|
||||
## Steps to Reproduce
|
||||
|
||||
1. ...
|
||||
2. ...
|
||||
3. ...
|
||||
|
||||
## Expected Behavior
|
||||
|
||||
What you expected to happen.
|
||||
|
||||
## Actual Behavior
|
||||
|
||||
What actually happened. Include the full error message or traceback if applicable.
|
||||
|
||||
```text
|
||||
Paste error output here
|
||||
```
|
||||
|
||||
## Environment
|
||||
|
||||
- OS: (e.g., macOS 15, Ubuntu 24.04, Windows 11)
|
||||
- Python version: (e.g., 3.12)
|
||||
- notebooklm-py version: (run `notebooklm --version`)
|
||||
- Install method: (pip, uv, pipx)
|
||||
- Surface: (CLI, Python API, MCP, REST server, desktop extension, docs)
|
||||
|
||||
## Debug Output
|
||||
|
||||
If applicable, run the failing command with `-vv` for verbose logging and paste the relevant output:
|
||||
|
||||
```bash
|
||||
notebooklm -vv <your-command-here>
|
||||
```
|
||||
|
||||
For auth/context issues, also include these outputs with cookies, emails, notebook
|
||||
titles, and paths redacted as needed:
|
||||
|
||||
```bash
|
||||
notebooklm doctor --json
|
||||
notebooklm status --paths --json
|
||||
notebooklm auth check --test --json
|
||||
```
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] I verified this bug exists on the latest version of notebooklm-py
|
||||
- [ ] I searched existing issues and this is not a duplicate
|
||||
@@ -0,0 +1 @@
|
||||
blank_issues_enabled: false
|
||||
@@ -0,0 +1,28 @@
|
||||
---
|
||||
name: Feature Request
|
||||
about: Suggest a new feature or enhancement
|
||||
title: ""
|
||||
labels: enhancement
|
||||
assignees: ""
|
||||
---
|
||||
|
||||
## Use Case
|
||||
|
||||
Describe the problem or workflow this feature would address.
|
||||
|
||||
## Target Surface
|
||||
|
||||
- [ ] CLI
|
||||
- [ ] Python API
|
||||
- [ ] MCP
|
||||
- [ ] REST server
|
||||
- [ ] Desktop extension
|
||||
- [ ] Documentation
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
How you'd like this to work (CLI usage, API example, etc.).
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
Any alternative approaches you've thought of.
|
||||
@@ -0,0 +1,23 @@
|
||||
## Summary
|
||||
|
||||
Brief description of the changes.
|
||||
|
||||
## Related Issue
|
||||
|
||||
Closes #<issue_number>
|
||||
|
||||
## Changes
|
||||
|
||||
- ...
|
||||
|
||||
## Test Plan
|
||||
|
||||
- [ ] I tested these changes locally
|
||||
- [ ] Tests pass (`uv run pytest --cov=src/notebooklm --cov-report=term-missing --cov-report=json:coverage.json --cov-fail-under=90`)
|
||||
- [ ] Linting and formatting pass (`uv run pre-commit run --all-files`)
|
||||
- [ ] Type checking passes (`uv run mypy src/notebooklm --ignore-missing-imports`)
|
||||
- [ ] If this PR changes architectural shape, an ADR has been added or updated.
|
||||
|
||||
## Notes
|
||||
|
||||
Any additional context, trade-offs, or design decisions.
|
||||
@@ -0,0 +1,48 @@
|
||||
# CodeQL configuration for notebooklm-py.
|
||||
#
|
||||
# The default Python query pack flags ``print(... <sensitive value> ...)`` as
|
||||
# "clear-text logging of sensitive information" whenever a value tagged as
|
||||
# secret reaches a print statement, even if the value passed through a
|
||||
# project-local sanitiser first. ``notebooklm._logging.scrub_secrets`` IS
|
||||
# such a sanitiser — it applies the same redaction pattern set used by
|
||||
# ``RedactingFilter`` on the logging pipeline (cookies, CSRF tokens, OAuth
|
||||
# bearers, Set-Cookie response headers, etc.) — but CodeQL's default
|
||||
# allow-list doesn't know about it.
|
||||
#
|
||||
# Rather than peppering the codebase with per-line ``# noqa`` comments,
|
||||
# disable the ``py/clear-text-logging-sensitive-data`` query for paths
|
||||
# where scrub_secrets is the load-bearing redactor. The query still runs
|
||||
# everywhere else in the codebase, so a new clear-text leak in
|
||||
# (say) ``src/notebooklm/_*.py`` would still surface.
|
||||
#
|
||||
# Scoped narrowly to the rpc-health canary because (a) it's the only
|
||||
# place where decoded RPC responses are printed for diagnostics, and
|
||||
# (b) that path already routes through scrub_secrets at every site —
|
||||
# see the comments at each print site in scripts/check_rpc_health.py.
|
||||
#
|
||||
# References:
|
||||
# * https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning
|
||||
# * https://codeql.github.com/codeql-query-help/python/py-clear-text-logging-sensitive-data/
|
||||
|
||||
# Exclude the rule globally. The standard CodeQL config schema for
|
||||
# ``query-filters`` does NOT accept a ``paths`` sub-key — attempting to
|
||||
# scope the suppression to a specific file via that field silently
|
||||
# evaluates to "no exclusion". Since the only Python sites that hit
|
||||
# ``py/clear-text-logging-sensitive-data`` in this codebase are the
|
||||
# scrub_secrets-wrapped print paths in ``scripts/check_rpc_health.py``
|
||||
# (the rpc-health canary script, which is the only place we deliberately
|
||||
# echo decoded RPC responses for diagnostics), and every such site is
|
||||
# already routed through ``scrub_secrets``, a global exclude is the
|
||||
# correct trade-off. The rule remains a noisy false-positive generator
|
||||
# for any sanitiser CodeQL doesn't recognise, and we don't have a custom
|
||||
# sanitiser-model query pack to teach it about scrub_secrets.
|
||||
query-filters:
|
||||
- exclude:
|
||||
id: py/clear-text-logging-sensitive-data
|
||||
paths-ignore:
|
||||
# scripts/check_rpc_health.py is a maintenance canary that prints
|
||||
# decoded RPC bodies for human diagnosis when the API drifts. Every
|
||||
# print site there calls scrub_secrets first. Belt-and-braces: also
|
||||
# mark the file as path-ignored so any future CodeQL rule that
|
||||
# similarly mis-models the local sanitiser won't gate the PR.
|
||||
- 'scripts/check_rpc_health.py'
|
||||
@@ -0,0 +1,33 @@
|
||||
version: 2
|
||||
updates:
|
||||
- package-ecosystem: "pip"
|
||||
directory: "/"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
open-pull-requests-limit: 5
|
||||
groups:
|
||||
dev-dependencies:
|
||||
patterns:
|
||||
- "pytest*"
|
||||
- "mypy"
|
||||
- "ruff"
|
||||
- "vcrpy"
|
||||
commit-message:
|
||||
prefix: "deps"
|
||||
|
||||
- package-ecosystem: "github-actions"
|
||||
directory: "/"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
# Group all GitHub Actions updates into a single PR per week. Each PR
|
||||
# touches multiple ``uses:`` lines, so grouping reduces review noise and
|
||||
# keeps SHA-pinned third-party actions (publish/testpypi/claude/rpc-health
|
||||
# /nightly/verify-package) moving together with their first-party
|
||||
# ``actions/*`` peers. See ``scripts/check_action_pinning.py`` for the
|
||||
# invariant Dependabot must keep honoring.
|
||||
groups:
|
||||
actions:
|
||||
patterns:
|
||||
- "*"
|
||||
commit-message:
|
||||
prefix: "ci"
|
||||
@@ -0,0 +1,65 @@
|
||||
name: Claude Code
|
||||
|
||||
on:
|
||||
issue_comment:
|
||||
types: [created]
|
||||
pull_request_review_comment:
|
||||
types: [created]
|
||||
issues:
|
||||
types: [opened, assigned]
|
||||
pull_request_review:
|
||||
types: [submitted]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
claude:
|
||||
# AND-gate on both the actor (``sender.login``) and the keyword. For the
|
||||
# ``issues`` event we additionally require ``issue.user.login`` to match
|
||||
# — otherwise an attacker could open an issue with ``@claude`` in the
|
||||
# body and wait for ``teng-lin`` to assign it: ``sender`` would then be
|
||||
# ``teng-lin`` while the prompt text remains attacker-authored. Keeping
|
||||
# the author check pins both the trigger and the prompt to the same
|
||||
# trusted account.
|
||||
if: |
|
||||
github.event.sender.login == 'teng-lin' && (
|
||||
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
|
||||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
|
||||
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
|
||||
(github.event_name == 'issues' && github.event.issue.user.login == 'teng-lin' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
|
||||
)
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write # Required to post inline review-thread comments (not just a sticky issue comment)
|
||||
issues: read
|
||||
id-token: write
|
||||
actions: read # Required for Claude to read CI results on PRs
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v7
|
||||
with:
|
||||
fetch-depth: 1
|
||||
persist-credentials: false
|
||||
|
||||
- name: Run Claude Code
|
||||
id: claude
|
||||
uses: anthropics/claude-code-action@558b1d6cab4085c7753fe402c10bef0fbb92ac7a # @ v1.0.165
|
||||
with:
|
||||
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
|
||||
|
||||
# This is an optional setting that allows Claude to read CI results on PRs
|
||||
additional_permissions: |
|
||||
actions: read
|
||||
|
||||
# Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it.
|
||||
# prompt: 'Update the pull request description to include a summary of changes.'
|
||||
|
||||
# Allow the GitHub inline-comment tool so a `@claude review` lands its
|
||||
# findings as inline PR review-thread comments (visible + addressable
|
||||
# like gemini/coderabbit), instead of only a single sticky issue
|
||||
# comment that the merge gate can miss. `--allowedTools` is additive in
|
||||
# this action (base Read/Glob/Grep/LS stay included). See docs/usage.md.
|
||||
claude_args: '--allowedTools "mcp__github_inline_comment__create_inline_comment"'
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
name: CodeQL
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
schedule:
|
||||
- cron: '0 12 * * 1' # Monday at noon UTC
|
||||
|
||||
jobs:
|
||||
analyze:
|
||||
name: Analyze
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
actions: read
|
||||
security-events: write
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Initialize CodeQL
|
||||
uses: github/codeql-action/init@v4
|
||||
with:
|
||||
languages: python
|
||||
# ``py/clear-text-logging-sensitive-data`` flags every path
|
||||
# where a value tagged as secret reaches a print statement,
|
||||
# even when the project's local sanitiser
|
||||
# ``notebooklm._logging.scrub_secrets`` has already redacted
|
||||
# the value. ``codeql-config.yml`` scopes the suppression
|
||||
# narrowly to ``scripts/check_rpc_health.py`` (the only
|
||||
# script that prints decoded RPC responses for diagnostics);
|
||||
# the query still runs everywhere else.
|
||||
config-file: .github/codeql-config.yml
|
||||
|
||||
- name: Perform CodeQL Analysis
|
||||
uses: github/codeql-action/analyze@v4
|
||||
@@ -0,0 +1,68 @@
|
||||
# Runs pip-audit against the locked environment so a CVE landing in a
|
||||
# transitive dep surfaces in CI rather than at next user install. Any
|
||||
# unresolved advisory fails the job (a hard merge gate on PRs, a red main
|
||||
# build on push/schedule).
|
||||
name: dependency-audit
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
paths:
|
||||
- "pyproject.toml"
|
||||
- "uv.lock"
|
||||
- ".github/workflows/dependency-audit.yml"
|
||||
schedule:
|
||||
# Nightly at 06:00 UTC so transitive CVEs surface even when no PR
|
||||
# touches the manifest.
|
||||
- cron: "0 6 * * *"
|
||||
workflow_dispatch: {}
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
pip-audit:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
enable-cache: true
|
||||
|
||||
- name: Sync locked env (browser + dev + markdown)
|
||||
# Retried to ride out transient registry/network blips: a real
|
||||
# lockfile problem fails all 3 attempts identically, so this only
|
||||
# absorbs flakes — it never masks a genuine resolution error.
|
||||
shell: bash
|
||||
run: |
|
||||
attempt=1; max=3
|
||||
until uv sync --frozen --extra browser --extra dev --extra markdown; do
|
||||
if [ "$attempt" -ge "$max" ]; then
|
||||
echo "::error::uv sync failed after $max attempts" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "::warning::uv sync attempt $attempt failed; retrying in $((attempt * 15))s" >&2
|
||||
sleep $((attempt * 15))
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
- name: Install pip-audit into the locked env
|
||||
# Pin pip-audit on the current major (2.x) so two consecutive nightly
|
||||
# runs use the same advisory-DB query logic — without this, a silent
|
||||
# bump to 3.x could change strictness without an accompanying PR.
|
||||
run: uv pip install 'pip-audit>=2.7.0,<3'
|
||||
|
||||
- name: pip-audit (locked env)
|
||||
# Hard merge gate: any unresolved advisory fails the job. If a
|
||||
# transient advisory genuinely needs to be ignored, pin the fix in
|
||||
# uv.lock or use pip-audit's --ignore-vuln for a tracked exception.
|
||||
# Audit the exported lock graph instead of the installed environment so
|
||||
# unreleased local package versions do not fail strict collection.
|
||||
run: |
|
||||
set -o pipefail
|
||||
uv export --frozen --extra browser --extra dev --extra markdown --format requirements-txt --no-emit-project \
|
||||
| uv run pip-audit --strict --require-hashes --disable-pip -r /dev/stdin
|
||||
@@ -0,0 +1,380 @@
|
||||
name: Nightly E2E Tests
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Main branch: 6 AM UTC (10 PM PST / 1 AM EST)
|
||||
- cron: '0 6 * * *'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
test_filter:
|
||||
description: 'Specific test to run (e.g., tests/e2e/test_downloads.py::TestDownloadReport)'
|
||||
required: false
|
||||
default: ''
|
||||
custom_branch:
|
||||
description: 'Override branch to test (leave empty to test the branch you triggered from)'
|
||||
required: false
|
||||
default: ''
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
# Determine which branch to test
|
||||
resolve-branch:
|
||||
runs-on: ubuntu-latest
|
||||
if: github.repository == 'teng-lin/notebooklm-py'
|
||||
outputs:
|
||||
branch: ${{ steps.resolve.outputs.branch }}
|
||||
is_standard: ${{ steps.resolve.outputs.is_standard }}
|
||||
sha: ${{ steps.target.outputs.sha }}
|
||||
short_sha: ${{ steps.target.outputs.short_sha }}
|
||||
steps:
|
||||
- name: Resolve target branch
|
||||
id: resolve
|
||||
# Route every workflow_dispatch input + GitHub-context value through
|
||||
# the env block so crafted values (e.g. ``"; rm -rf / ; #``) reach
|
||||
# the shell as literal strings, not as inlined script. ``GITHUB_OUTPUT``
|
||||
# is set automatically by the runner.
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
CUSTOM_BRANCH: ${{ inputs.custom_branch }}
|
||||
REF_NAME: ${{ github.ref_name }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "$EVENT_NAME" = "schedule" ]; then
|
||||
# Scheduled runs test main only. Release branches are manual.
|
||||
TARGET="main"
|
||||
else
|
||||
# Manual: use custom_branch if set, otherwise use triggering branch
|
||||
if [ -n "$CUSTOM_BRANCH" ]; then
|
||||
TARGET="$CUSTOM_BRANCH"
|
||||
else
|
||||
TARGET="$REF_NAME"
|
||||
fi
|
||||
fi
|
||||
echo "branch=$TARGET" >> "$GITHUB_OUTPUT"
|
||||
# Check if it's main or a release branch
|
||||
case "$TARGET" in
|
||||
main|release/*)
|
||||
echo "is_standard=true" >> "$GITHUB_OUTPUT"
|
||||
;;
|
||||
*)
|
||||
echo "is_standard=false" >> "$GITHUB_OUTPUT"
|
||||
;;
|
||||
esac
|
||||
echo "Resolved branch: $TARGET"
|
||||
|
||||
- uses: actions/checkout@v7
|
||||
if: steps.resolve.outputs.is_standard == 'true'
|
||||
with:
|
||||
ref: ${{ steps.resolve.outputs.branch }}
|
||||
fetch-depth: 1
|
||||
persist-credentials: false
|
||||
|
||||
- name: Resolve target commit
|
||||
id: target
|
||||
if: steps.resolve.outputs.is_standard == 'true'
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
SHA=$(git rev-parse HEAD)
|
||||
echo "sha=$SHA" >> "$GITHUB_OUTPUT"
|
||||
echo "short_sha=$(git rev-parse --short=12 HEAD)" >> "$GITHUB_OUTPUT"
|
||||
echo "Resolved commit: $SHA"
|
||||
|
||||
# Run E2E tests on the resolved branch
|
||||
e2e:
|
||||
name: E2E Tests (${{ needs.resolve-branch.outputs.branch }}@${{ needs.resolve-branch.outputs.short_sha }}/${{ matrix.os }})
|
||||
needs: resolve-branch
|
||||
if: needs.resolve-branch.outputs.is_standard == 'true'
|
||||
runs-on: ${{ matrix.os }}
|
||||
# Secret-bearing job. Two-layer gating:
|
||||
#
|
||||
# 1. ``needs.resolve-branch.outputs.is_standard`` — set by the upstream
|
||||
# ``resolve-branch`` job. Only ``main`` and ``release/*`` branches
|
||||
# (or scheduled cron triggers, which resolve to one of those) flip
|
||||
# this to ``true``; any other branch keeps it ``false`` and the
|
||||
# job-level ``if:`` above skips this whole job (no secret values
|
||||
# land in the runner env). The step-level ``if:`` guards on the
|
||||
# secret-bearing steps below are belt-and-suspenders so the gate
|
||||
# stays visible if the job-level guard is ever loosened.
|
||||
# 2. ``environment: protected-readonly`` (unconditional). The secrets
|
||||
# this job consumes (``NOTEBOOKLM_AUTH_JSON`` and friends) live only
|
||||
# in that environment, so the binding has to be unconditional —
|
||||
# issue #1009 surfaced the scheduled cron failing when the previous
|
||||
# conditional (``workflow_dispatch``-only) form fell back to
|
||||
# repo-level secrets that no longer exist. Add a ``required
|
||||
# reviewers`` rule on the environment if you want to block
|
||||
# workflow_dispatch behind manual approval; scheduled runs would
|
||||
# then queue too, so today this env carries no protection rules.
|
||||
# See docs/development.md → "Workflow secret gates".
|
||||
environment: protected-readonly
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [ubuntu-latest, windows-latest]
|
||||
env:
|
||||
# Pin Playwright's browser install dir to a single workspace-relative
|
||||
# path on every OS — see test.yml for the rationale (sidesteps the
|
||||
# actions/cache@v6 Windows dual-path restore flake).
|
||||
PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/.playwright-browsers
|
||||
# Single source of truth for the coverage-floor sentinel path, referenced by
|
||||
# both the E2E step (writes it) and the "Enforce coverage floors" step (gates
|
||||
# on it). Harmless when unset-enforcement steps run — they never write it.
|
||||
E2E_COVERAGE_FLOOR_SENTINEL: ${{ github.workspace }}/.e2e-coverage-floor-failed
|
||||
|
||||
concurrency:
|
||||
group: nightly-${{ needs.resolve-branch.outputs.branch }}-${{ matrix.os }}
|
||||
cancel-in-progress: true
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
ref: ${{ needs.resolve-branch.outputs.sha }}
|
||||
fetch-depth: 1
|
||||
persist-credentials: false
|
||||
|
||||
- name: Stamp E2E target
|
||||
env:
|
||||
EXPECTED_SHA: ${{ needs.resolve-branch.outputs.sha }}
|
||||
TARGET_BRANCH: ${{ needs.resolve-branch.outputs.branch }}
|
||||
MATRIX_OS: ${{ matrix.os }}
|
||||
shell: bash
|
||||
run: |
|
||||
ACTUAL_SHA=$(git rev-parse HEAD)
|
||||
if [ "$ACTUAL_SHA" != "$EXPECTED_SHA" ]; then
|
||||
echo "::error::Checked out $ACTUAL_SHA, expected $EXPECTED_SHA"
|
||||
exit 1
|
||||
fi
|
||||
{
|
||||
echo "### E2E target"
|
||||
echo "- Branch: \`$TARGET_BRANCH\`"
|
||||
echo "- Commit: \`$ACTUAL_SHA\`"
|
||||
echo "- OS: \`$MATRIX_OS\`"
|
||||
} >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
cache: 'pip'
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # @ v7.0.0
|
||||
|
||||
- name: Install dependencies
|
||||
# `uv sync --frozen` uses `uv.lock` for deterministic dep resolution
|
||||
# (same as the contributor workflow in docs/installation.md). Extras
|
||||
# cover the full E2E surface: `browser` = playwright, `dev` = pytest +
|
||||
# plugins, `markdown` = markdownify (used by source-conversion tests).
|
||||
# Retried to ride out transient registry/network blips: a real lockfile
|
||||
# problem fails all 3 attempts identically, so this only absorbs flakes
|
||||
# — it never masks a genuine resolution error.
|
||||
shell: bash
|
||||
run: |
|
||||
# --extra impersonate so the ubuntu curl_cffi smoke step below can run.
|
||||
# --extra mcp so the MCP/CLI live e2e suite (tests/e2e/test_mcp*.py,
|
||||
# test_cli_live.py) actually RUNS instead of importorskip-ping fastmcp.
|
||||
# --extra server so the REST live e2e suite (tests/e2e/test_server_live.py)
|
||||
# RUNS instead of importorskip-ping fastapi.
|
||||
attempt=1; max=3
|
||||
until uv sync --frozen --extra browser --extra dev --extra markdown --extra impersonate --extra mcp --extra server; do
|
||||
if [ "$attempt" -ge "$max" ]; then
|
||||
echo "::error::uv sync failed after $max attempts" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "::warning::uv sync attempt $attempt failed; retrying in $((attempt * 15))s" >&2
|
||||
sleep $((attempt * 15))
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
- name: Get Playwright version
|
||||
id: playwright-version
|
||||
shell: bash
|
||||
# `uv pip show` / bare `pip show` after activation can resolve outside
|
||||
# `.venv` when uv has not seeded pip into the project venv, yielding an
|
||||
# empty version that collapses the browser cache key to
|
||||
# `playwright-${{ matrix.os }}--ws`. `importlib.metadata` reads
|
||||
# `.venv`'s actual installed dist-info via the active interpreter, so
|
||||
# the version is always correct.
|
||||
run: |
|
||||
VERSION=$(uv run python -c 'import importlib.metadata as m; print(m.version("playwright"))')
|
||||
echo "version=$VERSION" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Cache Playwright browsers
|
||||
uses: actions/cache@v6
|
||||
with:
|
||||
path: ${{ github.workspace }}/.playwright-browsers
|
||||
# ``-ws`` suffix invalidates archives keyed to the prior dual-path
|
||||
# stanza so first restores after this change land on the new path.
|
||||
key: playwright-${{ matrix.os }}-${{ steps.playwright-version.outputs.version }}-ws
|
||||
|
||||
- name: Install Playwright browsers
|
||||
run: uv run playwright install chromium
|
||||
|
||||
- name: Install Playwright system dependencies (Linux)
|
||||
if: runner.os == 'Linux'
|
||||
run: uv run playwright install-deps chromium
|
||||
|
||||
# Fail-fast preflight. Without this, an empty NOTEBOOKLM_AUTH_JSON
|
||||
# (env-binding misconfig, missing repo-level fallback, etc.) lets the
|
||||
# E2E suite proceed; pytest then skips every auth-requiring test
|
||||
# silently and the job lands green with 0 tests run (issue #1009).
|
||||
- name: Verify auth secret is present
|
||||
if: needs.resolve-branch.outputs.is_standard == 'true'
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID }}
|
||||
shell: bash
|
||||
run: |
|
||||
if [ -z "${NOTEBOOKLM_AUTH_JSON:-}" ]; then
|
||||
echo "::error::NOTEBOOKLM_AUTH_JSON resolved to empty. Env binding or secret config is broken — see docs/development.md → Workflow secret gates."
|
||||
exit 1
|
||||
fi
|
||||
if [ -z "${NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID:-}" ]; then
|
||||
echo "::error::NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID resolved to empty."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Run E2E tests
|
||||
id: e2e
|
||||
# Hard gate: only standard branches (main/release/*, or scheduled cron
|
||||
# runs which resolve to one of those) expose credentials. Any other
|
||||
# branch — including ad-hoc workflow_dispatch on a feature branch —
|
||||
# gets ``is_standard == 'false'`` from resolve-branch and skips here
|
||||
# outright. The job-level ``environment:`` adds the maintainer approval
|
||||
# layer for workflow_dispatch runs that DO resolve as standard.
|
||||
if: needs.resolve-branch.outputs.is_standard == 'true'
|
||||
# Don't fail the job here — the retry step below gets a 10-min cool-down
|
||||
# shot at any failures, and its exit code is what marks the job red.
|
||||
continue-on-error: true
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID }}
|
||||
NOTEBOOKLM_GENERATION_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_GENERATION_NOTEBOOK_ID }}
|
||||
# Enforce the live coverage floors on the nightly only (fresh quota): if a
|
||||
# monitored surface (live chat / live generation) produced zero real coverage
|
||||
# because every marked test was rate-limited, record it to the sentinel that
|
||||
# the "Enforce coverage floors" step below gates on. Written (not raised) so
|
||||
# a pure-skip run stays exit 0 and doesn't trip the continue-on-error retry.
|
||||
# The release job (verify-package) leaves this unset → skip-only (#1819).
|
||||
# (E2E_COVERAGE_FLOOR_SENTINEL is defined once at job level.)
|
||||
E2E_ENFORCE_COVERAGE_FLOOR: "1"
|
||||
# Route every workflow_dispatch input + upstream needs.* value through
|
||||
# the env block so crafted values (e.g. ``"; echo PWN``) reach pytest
|
||||
# as literal strings, not as inlined shell. ``TARGET_BRANCH`` is
|
||||
# derived from ``inputs.custom_branch`` upstream; ``TEST_FILTER`` is
|
||||
# the direct workflow_dispatch input (empty for scheduled runs).
|
||||
TEST_FILTER: ${{ inputs.test_filter }}
|
||||
TARGET_BRANCH: ${{ needs.resolve-branch.outputs.branch }}
|
||||
MATRIX_OS: ${{ matrix.os }}
|
||||
shell: bash
|
||||
run: |
|
||||
echo "Testing branch: $TARGET_BRANCH"
|
||||
# Start from a clean coverage-floor sentinel so a stale file from a prior
|
||||
# run on a reused/self-hosted runner can't false-red the enforcement step.
|
||||
rm -f "$E2E_COVERAGE_FLOOR_SENTINEL"
|
||||
# --reruns 1 catches sub-minute network blips; longer chat-throttle
|
||||
# recovery is handled by the cool-down retry step below.
|
||||
if [ -n "$TEST_FILTER" ]; then
|
||||
# Run specific test(s) when filter provided.
|
||||
# ``--`` separator forces $TEST_FILTER to be parsed as positional
|
||||
# file/path args, never as pytest options (e.g. a crafted ``--co``
|
||||
# or ``-k "expr"`` value cannot inject pytest flags).
|
||||
# The coverage floor is a whole-surface guarantee — meaningless for a
|
||||
# single filtered test, and a marked-but-throttled pick would false-red
|
||||
# the run — so disable enforcement for filtered dispatches (codex).
|
||||
unset E2E_ENFORCE_COVERAGE_FLOOR
|
||||
uv run pytest -s -v --tb=short --reruns 1 --reruns-delay 30 -- "$TEST_FILTER"
|
||||
elif [ "$MATRIX_OS" = "ubuntu-latest" ]; then
|
||||
# Linux: run all E2E tests except variants
|
||||
uv run pytest tests/e2e -m "not variants" -s -v --tb=short --reruns 1 --reruns-delay 30
|
||||
else
|
||||
# Windows: run only read-only tests (safer, avoids write operations)
|
||||
uv run pytest tests/e2e -m "readonly and not variants" -s -v --tb=short --reruns 1 --reruns-delay 30
|
||||
fi
|
||||
|
||||
- name: Retry failed E2E tests after 10-min cool-down
|
||||
# Tests still throttled after the cool-down become skips via the
|
||||
# _install_chat_rate_limit_skip fixture in tests/e2e/conftest.py.
|
||||
# Same ``is_standard`` hard gate as the primary E2E step above — the
|
||||
# retry must never expose secrets on a non-standard branch even if a
|
||||
# previous step somehow ran (e.g. re-run with edited workflow inputs).
|
||||
if: |
|
||||
needs.resolve-branch.outputs.is_standard == 'true'
|
||||
&& steps.e2e.outcome == 'failure'
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID }}
|
||||
NOTEBOOKLM_GENERATION_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_GENERATION_NOTEBOOK_ID }}
|
||||
# Enforce here too: a marked test that failed on the main run and turns into
|
||||
# a rate-limit skip after the cool-down would otherwise leave no PASS event
|
||||
# for its surface. Appends PASS/SKIP events to the same job-level sentinel
|
||||
# (the main step's rm -f already cleared any stale one). (codex/coderabbit)
|
||||
E2E_ENFORCE_COVERAGE_FLOOR: "1"
|
||||
shell: bash
|
||||
run: |
|
||||
# Missing lastfailed after a failed e2e step means pytest crashed
|
||||
# before writing the cache (import error, OOM, etc.) — that's a real
|
||||
# failure, not a recoverable rate-limit, so fail the job rather than
|
||||
# silently going green.
|
||||
if [ ! -f .pytest_cache/v/cache/lastfailed ]; then
|
||||
echo "::error::No lastfailed cache from previous step; pytest likely crashed before running tests."
|
||||
exit 1
|
||||
fi
|
||||
echo "Initial E2E run failed. Sleeping 600s before retrying failed tests."
|
||||
sleep 600
|
||||
# Pin ``tests/e2e`` so pyproject's ``addopts = --ignore=tests/e2e``
|
||||
# doesn't drop every lastfailed entry; otherwise pytest's default
|
||||
# ``--last-failed-no-failures=all`` would expand to the full non-e2e
|
||||
# suite under NOTEBOOKLM_AUTH_JSON. ``=none`` makes a missing cache
|
||||
# fail fast instead.
|
||||
uv run pytest tests/e2e --last-failed --last-failed-no-failures=none -s -v --tb=short
|
||||
|
||||
- name: curl_cffi transport smoke (live, minimal)
|
||||
# Minimum live coverage of the opt-in curl_cffi transport: the handful of
|
||||
# @pytest.mark.impersonate_smoke e2e tests (one per transport-critical op —
|
||||
# read / ask / upload / download) run with NOTEBOOKLM_TRANSPORT=curl_cffi.
|
||||
# ubuntu-only (the write path exercises the upload-fingerprint case that no
|
||||
# offline test can reach; Windows just needs the read/ask path, already
|
||||
# covered by the httpx legs + the test.yml hermetic matrix). Same is_standard
|
||||
# secret gate as the steps above.
|
||||
if: |
|
||||
needs.resolve-branch.outputs.is_standard == 'true'
|
||||
&& matrix.os == 'ubuntu-latest'
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID }}
|
||||
NOTEBOOKLM_GENERATION_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_GENERATION_NOTEBOOK_ID }}
|
||||
NOTEBOOKLM_TRANSPORT: curl_cffi
|
||||
shell: bash
|
||||
run: uv run pytest tests/e2e -m impersonate_smoke -s -v --tb=short --reruns 1 --reruns-delay 30
|
||||
|
||||
- name: Enforce coverage floors
|
||||
# Gate the nightly on the live coverage floors independently of the
|
||||
# continue-on-error main run + --last-failed retry (which can't re-attempt
|
||||
# throttled generation — those are skips, not failures). The main e2e step
|
||||
# records a breach to the sentinel; fail here if any surface produced zero
|
||||
# real coverage (every marked live-chat / live-generation test rate-limited).
|
||||
# ``always()`` so a breach recorded before an unrelated later failure still
|
||||
# surfaces. The release job never sets the sentinel, so this is nightly-only.
|
||||
if: ${{ always() && needs.resolve-branch.outputs.is_standard == 'true' }}
|
||||
shell: bash
|
||||
run: |
|
||||
sentinel="$E2E_COVERAGE_FLOOR_SENTINEL"
|
||||
if [ ! -f "$sentinel" ]; then
|
||||
echo "Coverage floors satisfied (no monitored surface emitted a skip)."
|
||||
exit 0
|
||||
fi
|
||||
# Reconcile PASS/SKIP events accumulated across the main run + retry: a
|
||||
# surface breaches only if it was SKIP-ped somewhere and PASSed nowhere.
|
||||
passed="$(awk -F'\t' '$1=="PASS"{print $2}' "$sentinel" | sort -u)"
|
||||
skipped="$(awk -F'\t' '$1=="SKIP"{print $2}' "$sentinel" | sort -u)"
|
||||
breach="$(comm -23 <(printf '%s\n' "$skipped" | sed '/^$/d') \
|
||||
<(printf '%s\n' "$passed" | sed '/^$/d'))"
|
||||
if [ -n "$breach" ]; then
|
||||
echo "::error::E2E coverage floor breached — surface(s) rate-limited with zero real coverage:"
|
||||
printf ' %s\n' "$breach"
|
||||
exit 1
|
||||
fi
|
||||
echo "Coverage floors satisfied (every skipped surface also had a passing test)."
|
||||
@@ -0,0 +1,95 @@
|
||||
# Publish the remote MCP server image to Docker Hub on a release tag.
|
||||
#
|
||||
# Opt-in: only runs on the canonical repo AND when the `DOCKERHUB_IMAGE`
|
||||
# repository variable is set (e.g. `youruser/notebooklm-mcp`). Configure once:
|
||||
# - Variables → DOCKERHUB_IMAGE = <namespace>/notebooklm-mcp
|
||||
# - Secrets → DOCKERHUB_USERNAME, DOCKERHUB_TOKEN (a Docker Hub access token)
|
||||
# Forks and unconfigured repos skip it entirely (no failing release job).
|
||||
#
|
||||
# Builds deploy/Dockerfile from the tagged source (context = repo root, same as
|
||||
# `docker compose`), multi-arch (amd64 + arm64). `:latest` only moves on a final
|
||||
# release — PEP 440 pre-releases (v0.8.0a1/b1/rc1) publish their version tag only.
|
||||
name: Publish MCP image to Docker Hub
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- "v*"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
publish-image:
|
||||
name: Build and push notebooklm-mcp
|
||||
runs-on: ubuntu-latest
|
||||
# `release` environment gates the Docker Hub secrets (maintainer-approved,
|
||||
# same as publish.yml). The actor pin restricts publishing to the
|
||||
# maintainer's own tag push; the repo + var checks skip forks/unconfigured
|
||||
# repos so a release never fails on a missing Docker Hub config.
|
||||
environment: release
|
||||
if: ${{ github.repository == 'teng-lin/notebooklm-py' && github.actor == 'teng-lin' && vars.DOCKERHUB_IMAGE != '' }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
# Full history so ALL tags are present — we compare against every released
|
||||
# version to decide whether :latest moves (never backward to a maintenance
|
||||
# release). `.dockerignore` excludes `.git`, so this never bloats the image.
|
||||
fetch-depth: 0
|
||||
|
||||
# Pin Python so `tomllib` (3.11+) in the next step never depends on the
|
||||
# runner's default python3 — mirrors publish.yml.
|
||||
- uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Derive image tags (and validate tag == pyproject version)
|
||||
id: meta
|
||||
run: |
|
||||
VERSION=${GITHUB_REF#refs/tags/v}
|
||||
TOML_VERSION=$(python3 -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
|
||||
if [ "$VERSION" != "$TOML_VERSION" ]; then
|
||||
echo "Error: tag version ($VERSION) doesn't match pyproject.toml ($TOML_VERSION)" >&2
|
||||
exit 1
|
||||
fi
|
||||
IMAGE="${{ vars.DOCKERHUB_IMAGE }}"
|
||||
TAGS="${IMAGE}:${VERSION}"
|
||||
# :latest must point at the HIGHEST released version and never move
|
||||
# backward. Two gates: (1) a final release only — bare X.Y.Z, no PEP 440
|
||||
# pre-release suffix (a/b/rc); (2) it's the max stable tag — maintenance
|
||||
# releases on an older line are tagged directly (e.g. v0.7.4 after v0.8.0),
|
||||
# so those keep :latest where it is.
|
||||
if echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+$'; then
|
||||
HIGHEST=$(git tag --list 'v[0-9]*' | sed 's/^v//' \
|
||||
| grep -E '^[0-9]+\.[0-9]+\.[0-9]+$' | sort -V | tail -n1)
|
||||
if [ "$VERSION" = "$HIGHEST" ]; then
|
||||
TAGS="${TAGS},${IMAGE}:latest"
|
||||
else
|
||||
echo "Not the highest stable tag (highest=$HIGHEST) — publishing version tag only, :latest unchanged."
|
||||
fi
|
||||
fi
|
||||
echo "tags=${TAGS}" >> "$GITHUB_OUTPUT"
|
||||
echo "Publishing: ${TAGS}"
|
||||
|
||||
- uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # @ v4.2.0
|
||||
- uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # @ v4.2.0
|
||||
|
||||
- uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # @ v4.4.0
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
- uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # @ v7.3.0
|
||||
with:
|
||||
context: .
|
||||
file: deploy/Dockerfile
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: org.opencontainers.image.source=https://github.com/teng-lin/notebooklm-py
|
||||
@@ -0,0 +1,155 @@
|
||||
# Build the release bundles — the Claude Desktop `.mcpb` and the
|
||||
# Claude-uploadable skill archive (`notebooklm-skill.zip`, for chat / Cowork
|
||||
# Settings -> Capabilities) — and attach them to the GitHub Release.
|
||||
#
|
||||
# Why `release: published` (not `push: tags`)?
|
||||
# The release process (docs/releasing.md) creates the GitHub Release by hand
|
||||
# AFTER the tag is pushed and PyPI has published — there is no release to
|
||||
# attach an asset to at tag-push time, and a workflow that created one would
|
||||
# collide with the maintainer's manual `gh release create`. Reacting to
|
||||
# `release: published` fires exactly when that manual step completes, so
|
||||
# `gh release upload` always has a target.
|
||||
#
|
||||
# Stable AND pre-releases both ship a bundle. The launcher (`run_server.py`)
|
||||
# pins its exact version from the bundled `manifest.json` for a `vX.Y.ZaN`
|
||||
# bundle (an explicit `==` pre-release pin resolves under uv's default resolver
|
||||
# — no `--prerelease` flag, which would loosen transitive deps too), so a
|
||||
# pre-release bundle launches that pinned pre-release instead of silently
|
||||
# resolving the latest *stable* server. A stable bundle stays unpinned.
|
||||
#
|
||||
# Two jobs by design (token isolation, mirrors publish.yml): the `build` job
|
||||
# runs the third-party npm packer (and the pipx-built `skill package` CLI)
|
||||
# with NO release-write token, uploads the bundles as workflow artifacts; the
|
||||
# minimal `attach` job carries `contents: write` and runs only first-party
|
||||
# download + `gh release upload`. Splitting across jobs means nothing the
|
||||
# packer does (e.g. persisting a fake `gh` onto `$GITHUB_PATH`, or
|
||||
# `$GITHUB_ENV`) can reach the write token — a same-job
|
||||
# `env: GITHUB_TOKEN: ""` on the pack step would not stop that.
|
||||
name: Attach release bundles (.mcpb, skill zip)
|
||||
|
||||
on:
|
||||
release:
|
||||
types: [published]
|
||||
|
||||
# Serialize per-tag so a rare double-publish can't race two uploads onto the
|
||||
# same asset (mirrors publish.yml / publish-docker.yml). Never cancel a run
|
||||
# mid-upload.
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
# Top-level scope is read-only (the default-permissive GITHUB_TOKEN blast
|
||||
# radius is the thing check_workflow_permissions.py guards against). Only the
|
||||
# `attach` job below elevates to `contents: write`.
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
build:
|
||||
name: Pack the release bundles
|
||||
runs-on: ubuntu-latest
|
||||
# Only the canonical repo ships the bundle (forks' manifest URLs point
|
||||
# here anyway); both stable and pre-releases (see the version note above).
|
||||
if: ${{ github.repository == 'teng-lin/notebooklm-py' }}
|
||||
# No contents:write here: this job runs the third-party mcpb packer, so it
|
||||
# stays read-only. Even a compromised packer only sees a read-only token.
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
# Pack the extension from the tagged source, not the default branch,
|
||||
# so the bundle matches exactly what was released.
|
||||
ref: ${{ github.event.release.tag_name }}
|
||||
|
||||
- uses: actions/setup-node@v6
|
||||
with:
|
||||
node-version: "20"
|
||||
|
||||
- uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Guard — manifest version matches the release tag and pyproject
|
||||
# Defence in depth: the .mcpb we attach must carry the released
|
||||
# version. The tag (vX.Y.Z), pyproject.toml, and the bundle manifest
|
||||
# must all agree, or we abort before packing a mislabelled bundle.
|
||||
# Derive the version from the release's tag_name (not GITHUB_REF): on a
|
||||
# `release` event GITHUB_REF can reflect the release target branch
|
||||
# rather than the tag, whereas tag_name is unambiguous — and it matches
|
||||
# the checkout/upload steps.
|
||||
env:
|
||||
TAG_NAME: ${{ github.event.release.tag_name }}
|
||||
run: |
|
||||
TAG_VERSION="${TAG_NAME#v}"
|
||||
TOML_VERSION=$(python3 -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
|
||||
MANIFEST_VERSION=$(python3 -c "import json; print(json.load(open('desktop-extension/manifest.json'))['version'])")
|
||||
echo "tag=$TAG_VERSION pyproject=$TOML_VERSION manifest=$MANIFEST_VERSION"
|
||||
if [ "$TAG_VERSION" != "$TOML_VERSION" ] || [ "$TAG_VERSION" != "$MANIFEST_VERSION" ]; then
|
||||
echo "::error::version mismatch — tag=$TAG_VERSION pyproject=$TOML_VERSION manifest=$MANIFEST_VERSION" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Build the .mcpb bundle
|
||||
# `pack <source-dir> <output>` zips manifest.json + run_server.py and
|
||||
# validates the manifest against the DXT schema. `--yes` skips the npx
|
||||
# install prompt on a cold runner. The CLI is version-pinned (not
|
||||
# `@latest`) so the released bundle is reproducible from the tag and a
|
||||
# compromised/breaking upstream publish can't drift silently — bump
|
||||
# deliberately, like the SHA-pinned actions.
|
||||
run: npx --yes @anthropic-ai/mcpb@2.1.2 pack desktop-extension notebooklm-mcp.mcpb
|
||||
|
||||
- name: Build the skill archive
|
||||
# `pipx run --spec .` builds the tagged checkout into an isolated venv
|
||||
# (pipx is preinstalled on ubuntu-latest) and runs the `notebooklm`
|
||||
# console script — same read-only job as the mcpb packer, so the
|
||||
# pip-installed dependency tree never sees a release-write token.
|
||||
run: pipx run --spec . notebooklm skill package --output notebooklm-skill.zip
|
||||
|
||||
- name: Upload the bundle as a workflow artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # @ v7
|
||||
with:
|
||||
name: notebooklm-mcp-mcpb
|
||||
path: notebooklm-mcp.mcpb
|
||||
if-no-files-found: error
|
||||
retention-days: 1
|
||||
|
||||
- name: Upload the skill archive as a workflow artifact
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # @ v7
|
||||
with:
|
||||
name: notebooklm-skill-zip
|
||||
path: notebooklm-skill.zip
|
||||
if-no-files-found: error
|
||||
retention-days: 1
|
||||
|
||||
attach:
|
||||
name: Attach the bundles to the release
|
||||
needs: build
|
||||
runs-on: ubuntu-latest
|
||||
# Minimal write-scoped job: only first-party download + `gh release upload`
|
||||
# run here — no checkout, no node, no third-party code — so the
|
||||
# release-write token has no attacker-controllable co-tenant (mirrors the
|
||||
# split in publish.yml). Skipped automatically when `build` is skipped.
|
||||
permissions:
|
||||
contents: write
|
||||
|
||||
steps:
|
||||
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # @ v8.0.1
|
||||
with:
|
||||
name: notebooklm-mcp-mcpb
|
||||
|
||||
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # @ v8.0.1
|
||||
with:
|
||||
name: notebooklm-skill-zip
|
||||
|
||||
- name: Attach the bundles to the release
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
# This job has no checkout, so `gh` has no local git remote to infer
|
||||
# the repo from — give it the repo explicitly or the upload fails.
|
||||
GH_REPO: ${{ github.repository }}
|
||||
# Route tag_name through env (not inline `${{ }}` in run:) so a crafted
|
||||
# tag can't inject shell into this contents:write step.
|
||||
TAG_NAME: ${{ github.event.release.tag_name }}
|
||||
# `--clobber` makes re-runs idempotent (replaces an existing asset of
|
||||
# the same name instead of erroring).
|
||||
run: gh release upload "$TAG_NAME" notebooklm-mcp.mcpb notebooklm-skill.zip --clobber
|
||||
@@ -0,0 +1,122 @@
|
||||
name: Publish to PyPI
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- "v*"
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
build-and-test:
|
||||
name: Build wheel and run release smoke
|
||||
runs-on: ubuntu-latest
|
||||
# Smoke install pulls `[browser,dev,markdown]` extras (pytest, playwright,
|
||||
# mypy, ruff, markdownify, transitive deps). A compromised dep here is a
|
||||
# supply-chain concern but cannot mint a Trusted Publishing OIDC token,
|
||||
# because this job is not granted `id-token: write`. The publish job
|
||||
# below carries that scope and runs nothing except the trusted PyPA
|
||||
# action. See issue #820 for the threat model.
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Validate tag matches version
|
||||
run: |
|
||||
TAG_VERSION=${GITHUB_REF#refs/tags/v}
|
||||
TOML_VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml', 'rb'))['project']['version'])")
|
||||
if [ "$TAG_VERSION" != "$TOML_VERSION" ]; then
|
||||
echo "Error: Tag version ($TAG_VERSION) doesn't match pyproject.toml ($TOML_VERSION)"
|
||||
exit 1
|
||||
fi
|
||||
echo "Version validated: $TOML_VERSION"
|
||||
|
||||
- name: Install build tools
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install build==1.5.0
|
||||
|
||||
- name: Build package
|
||||
run: |
|
||||
python -m build
|
||||
|
||||
- name: Upload distribution artifacts
|
||||
# Capture pristine wheel + sdist BEFORE any third-party code (smoke
|
||||
# install, playwright, pytest) runs against `dist/`. If we uploaded
|
||||
# after the smoke install, a compromised dev/test dep could mutate
|
||||
# `dist/` post-build, and the publish job would upload tampered bytes
|
||||
# — attested with the project's trusted OIDC identity. Uploading here
|
||||
# snapshots the build output before any attacker-controlled code has
|
||||
# touched the runner. Smoke install/test below still runs against
|
||||
# `dist/` for behavior validation but can no longer influence what
|
||||
# the publish job receives.
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # @ v7
|
||||
with:
|
||||
name: notebooklm-py-dist
|
||||
path: dist/
|
||||
if-no-files-found: error
|
||||
# One-day retention is intentional: publish consumes this artifact
|
||||
# immediately. See issue #829 for the recovery trade-off.
|
||||
retention-days: 1
|
||||
|
||||
- name: Install built wheel + canonical contributor extras in a clean venv
|
||||
# Canonical contributor extras (`[browser,dev,markdown]`, equivalent to
|
||||
# `[all]`) match the install path documented in `docs/installation.md`,
|
||||
# so the release smoke exercises the same surface area users will see.
|
||||
# Plain `[dev]` skipped playwright + markdownify, masking packaging bugs
|
||||
# in `[browser]`/`[markdown]` until users hit them post-release.
|
||||
run: |
|
||||
python -m venv smoke-venv
|
||||
WHEEL=$(ls dist/notebooklm_py-*.whl)
|
||||
smoke-venv/bin/pip install "${WHEEL}[browser,dev,markdown]"
|
||||
smoke-venv/bin/python -c "import notebooklm; print(notebooklm.__version__)"
|
||||
|
||||
- name: Install Playwright browser for smoke
|
||||
# `[browser]` only installs the playwright Python package; the chromium
|
||||
# binary + Linux system libs are needed for the unit-test smoke
|
||||
# (tests/unit/test_windows_compatibility.py drives sync_playwright()).
|
||||
run: |
|
||||
smoke-venv/bin/playwright install chromium
|
||||
smoke-venv/bin/playwright install-deps chromium
|
||||
|
||||
- name: Run unit tests against wheel
|
||||
run: smoke-venv/bin/pytest tests/unit -q --no-cov -x
|
||||
|
||||
publish:
|
||||
name: Publish to PyPI
|
||||
needs: build-and-test
|
||||
runs-on: ubuntu-latest
|
||||
environment: release
|
||||
# Minimum-scope publish job: only the trusted PyPA action runs here. No
|
||||
# `actions/checkout`, no Python install, no third-party dependencies in
|
||||
# the runner environment — so the OIDC token request that `id-token:
|
||||
# write` enables has no co-tenant code that could exfiltrate it.
|
||||
permissions:
|
||||
id-token: write
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Download distribution artifacts
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # @ v8.0.1
|
||||
with:
|
||||
name: notebooklm-py-dist
|
||||
path: dist/
|
||||
|
||||
- name: Publish to PyPI
|
||||
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # @ v1.14.0
|
||||
with:
|
||||
# PEP 740 attestations: pypa/gh-action-pypi-publish generates an
|
||||
# in-toto attestation for the uploaded artifacts and publishes it
|
||||
# via PyPI's Trusted Publishing flow (OIDC, no API token).
|
||||
attestations: true
|
||||
@@ -0,0 +1,424 @@
|
||||
name: RPC Health Check
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Main branch: 7 AM UTC daily (1 hour after main nightly E2E tests)
|
||||
- cron: '0 7 * * *'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
custom_branch:
|
||||
description: 'Override branch to test (leave empty to test the branch you triggered from)'
|
||||
required: false
|
||||
default: ''
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
# Determine which branch to test
|
||||
resolve-branch:
|
||||
runs-on: ubuntu-latest
|
||||
if: github.repository == 'teng-lin/notebooklm-py'
|
||||
outputs:
|
||||
branch: ${{ steps.resolve.outputs.branch }}
|
||||
is_standard: ${{ steps.resolve.outputs.is_standard }}
|
||||
steps:
|
||||
- name: Resolve target branch
|
||||
id: resolve
|
||||
env:
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
CUSTOM_BRANCH: ${{ inputs.custom_branch }}
|
||||
REF_NAME: ${{ github.ref_name }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "$EVENT_NAME" = "schedule" ]; then
|
||||
# Scheduled runs test main only. Release branches are manual.
|
||||
TARGET="main"
|
||||
else
|
||||
if [ -n "$CUSTOM_BRANCH" ]; then
|
||||
TARGET="$CUSTOM_BRANCH"
|
||||
else
|
||||
TARGET="$REF_NAME"
|
||||
fi
|
||||
fi
|
||||
echo "branch=$TARGET" >> "$GITHUB_OUTPUT"
|
||||
case "$TARGET" in
|
||||
main|release/*)
|
||||
echo "is_standard=true" >> "$GITHUB_OUTPUT"
|
||||
;;
|
||||
*)
|
||||
echo "is_standard=false" >> "$GITHUB_OUTPUT"
|
||||
;;
|
||||
esac
|
||||
echo "Resolved branch: $TARGET"
|
||||
|
||||
health-check:
|
||||
name: RPC Health Check (${{ needs.resolve-branch.outputs.branch }})
|
||||
needs: resolve-branch
|
||||
if: needs.resolve-branch.outputs.is_standard == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
concurrency:
|
||||
group: rpc-health-${{ needs.resolve-branch.outputs.branch }}
|
||||
cancel-in-progress: true
|
||||
# Secret-bearing job. ``NOTEBOOKLM_AUTH_JSON`` and friends live only in
|
||||
# the ``protected-readonly`` GitHub Environment (issue #1009: relying on
|
||||
# repo-level fallback for the scheduled-cron branch broke when those
|
||||
# secrets were migrated env-only). Binding the env unconditionally is
|
||||
# what gives every trigger — scheduled cron, workflow_dispatch from a
|
||||
# maintainer, etc. — access to the same secret values. Workflow_dispatch
|
||||
# is still hard-gated by ``needs.resolve-branch.outputs.is_standard``
|
||||
# above so a feature-branch dispatch never reaches this job at all.
|
||||
# Add a ``required reviewers`` rule on the environment if you want to
|
||||
# block workflow_dispatch behind manual approval; scheduled runs would
|
||||
# then queue too, so today this env carries no protection rules.
|
||||
# See docs/development.md → "Workflow secret gates".
|
||||
environment: protected-readonly
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
ref: ${{ needs.resolve-branch.outputs.branch }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
cache: 'pip'
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # @ v7.0.0
|
||||
|
||||
- name: Install dependencies
|
||||
# `uv sync --frozen` reproduces the lockfile-pinned dep tree. The RPC
|
||||
# health script (`scripts/check_rpc_health.py`) only needs the core
|
||||
# runtime deps — no `[browser]`/`[dev]`/`[markdown]` extras required.
|
||||
# Retried to ride out transient registry/network blips: a real lockfile
|
||||
# problem fails all 3 attempts identically, so this only absorbs flakes
|
||||
# — it never masks a genuine resolution error.
|
||||
shell: bash
|
||||
run: |
|
||||
attempt=1; max=3
|
||||
until uv sync --frozen; do
|
||||
if [ "$attempt" -ge "$max" ]; then
|
||||
echo "::error::uv sync failed after $max attempts" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "::warning::uv sync attempt $attempt failed; retrying in $((attempt * 15))s" >&2
|
||||
sleep $((attempt * 15))
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
# Fail-fast preflight. The script itself rejects empty
|
||||
# NOTEBOOKLM_AUTH_JSON downstream, but checking at the workflow layer
|
||||
# surfaces ``::error::`` annotations linked to the secret-config
|
||||
# misconfig before the ``continue-on-error`` step swallows the exit
|
||||
# code (issue #1009).
|
||||
- name: Verify auth secret is present
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID }}
|
||||
shell: bash
|
||||
run: |
|
||||
if [ -z "${NOTEBOOKLM_AUTH_JSON:-}" ]; then
|
||||
echo "::error::NOTEBOOKLM_AUTH_JSON resolved to empty. Env binding or secret config is broken — see docs/development.md → Workflow secret gates."
|
||||
exit 1
|
||||
fi
|
||||
if [ -z "${NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID:-}" ]; then
|
||||
echo "::error::NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID resolved to empty."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Run RPC Health Check
|
||||
id: health
|
||||
continue-on-error: true
|
||||
shell: bash
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID }}
|
||||
NOTEBOOKLM_GENERATION_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_GENERATION_NOTEBOOK_ID }}
|
||||
TARGET_BRANCH: ${{ needs.resolve-branch.outputs.branch }}
|
||||
run: |
|
||||
echo "Testing branch: $TARGET_BRANCH"
|
||||
set +e
|
||||
uv run python scripts/check_rpc_health.py --full 2>&1 | tee health-report.txt
|
||||
exit_code=${PIPESTATUS[0]}
|
||||
echo "exit_code=${exit_code}" >> "$GITHUB_OUTPUT"
|
||||
exit "$exit_code"
|
||||
|
||||
- name: Scrub secrets from health-report.txt
|
||||
# Defence-in-depth: the script itself avoids printing credential-shaped
|
||||
# data, but raw response bodies and exception tracebacks can still
|
||||
# surface session/CSRF/cookie material on regressions. We re-run the
|
||||
# report through ``notebooklm._logging.scrub_secrets`` before any
|
||||
# downstream consumer reads it (Add Summary, Create Issue on …,
|
||||
# Upload Report). Idempotent — safe to re-apply on subsequent runs.
|
||||
if: always()
|
||||
shell: bash
|
||||
run: |
|
||||
if [ ! -f health-report.txt ]; then
|
||||
echo "::warning::health-report.txt not produced; skipping scrub."
|
||||
exit 0
|
||||
fi
|
||||
uv run python - <<'PY'
|
||||
from pathlib import Path
|
||||
from notebooklm._logging import scrub_secrets
|
||||
|
||||
report = Path("health-report.txt")
|
||||
original = report.read_text(encoding="utf-8", errors="replace")
|
||||
scrubbed = scrub_secrets(original)
|
||||
if scrubbed != original:
|
||||
report.write_text(scrubbed, encoding="utf-8")
|
||||
# ``abs`` because replacement tokens (``***``) may be longer
|
||||
# OR shorter than the redacted secret depending on the
|
||||
# pattern, so the raw delta can be negative; the magnitude
|
||||
# is what's interesting for sanity-checking the scrub.
|
||||
print(
|
||||
f"Scrubbed health-report.txt "
|
||||
f"(delta {abs(len(original) - len(scrubbed))} chars)."
|
||||
)
|
||||
else:
|
||||
print("No secret-shaped substrings detected in health-report.txt.")
|
||||
PY
|
||||
|
||||
- name: Add Summary
|
||||
if: always()
|
||||
shell: bash
|
||||
run: |
|
||||
echo "## RPC Health Check Results" >> "$GITHUB_STEP_SUMMARY"
|
||||
grep -A 10 "^SUMMARY$" health-report.txt >> "$GITHUB_STEP_SUMMARY" || echo "No summary found" >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
# Bundle-drift monitor (capture_rpc_registry.py). NOT a PR gate — it depends
|
||||
# on Google's live external JS bundle, which can rotate at any time — so it
|
||||
# rides the nightly/issue-filing track here. ``--check`` gates id rotation
|
||||
# (ABSENT), ``--check-enums`` gates studio enum CHANGED/STALE (a selectable
|
||||
# format renumbered/retired — the #1597 class). NEW/UNPARSED, quota codes and
|
||||
# proto assertions print for visibility but never fail. ``continue-on-error``
|
||||
# so a drift hit files an issue rather than red-failing the canary outright.
|
||||
- name: Run bundle drift monitor
|
||||
id: bundle
|
||||
continue-on-error: true
|
||||
shell: bash
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
run: |
|
||||
set +e
|
||||
uv run python scripts/capture_rpc_registry.py --check --check-enums 2>&1 \
|
||||
| tee bundle-drift-report.txt
|
||||
exit_code=${PIPESTATUS[0]}
|
||||
echo "exit_code=${exit_code}" >> "$GITHUB_OUTPUT"
|
||||
exit "$exit_code"
|
||||
|
||||
- name: Scrub secrets from bundle-drift-report.txt
|
||||
# The bundle is public CDN content, but the discovery homepage read is
|
||||
# authenticated and a transport error could echo a request URL; scrub
|
||||
# before any downstream consumer reads the file (same posture as the
|
||||
# health-report scrub above). Idempotent.
|
||||
if: always()
|
||||
shell: bash
|
||||
run: |
|
||||
if [ ! -f bundle-drift-report.txt ]; then
|
||||
echo "::warning::bundle-drift-report.txt not produced; skipping scrub."
|
||||
exit 0
|
||||
fi
|
||||
uv run python - <<'PY'
|
||||
from pathlib import Path
|
||||
from notebooklm._logging import scrub_secrets
|
||||
|
||||
report = Path("bundle-drift-report.txt")
|
||||
original = report.read_text(encoding="utf-8", errors="replace")
|
||||
scrubbed = scrub_secrets(original)
|
||||
if scrubbed != original:
|
||||
report.write_text(scrubbed, encoding="utf-8")
|
||||
print(f"Scrubbed bundle-drift-report.txt (delta {abs(len(original) - len(scrubbed))} chars).")
|
||||
else:
|
||||
print("No secret-shaped substrings detected in bundle-drift-report.txt.")
|
||||
PY
|
||||
|
||||
- name: Check for existing bundle drift issue
|
||||
if: steps.bundle.outputs.exit_code != '0'
|
||||
id: dup_bundle
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
if ! count=$(gh issue list --repo "$GITHUB_REPOSITORY" \
|
||||
--state open --label automated --label rpc-breakage \
|
||||
--search 'in:title "Studio enum / RPC drift detected"' \
|
||||
--json number --jq 'length'); then
|
||||
echo "::warning::Bundle drift dedup probe failed; allowing issue creation"
|
||||
count=0
|
||||
fi
|
||||
echo "open=${count}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Create Issue on bundle drift
|
||||
if: |
|
||||
steps.bundle.outputs.exit_code != '0'
|
||||
&& steps.dup_bundle.outputs.open == '0'
|
||||
uses: peter-evans/create-issue-from-file@fca9117c27cdc29c6c4db3b86c48e4115a786710 # @ v6.0.0
|
||||
with:
|
||||
title: "Studio enum / RPC drift detected"
|
||||
content-filepath: bundle-drift-report.txt
|
||||
labels: rpc-breakage, automated
|
||||
|
||||
- name: Upload bundle drift report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: rpc-bundle-drift-report
|
||||
path: bundle-drift-report.txt
|
||||
retention-days: 30
|
||||
|
||||
# Each issue-creating step is paired with a dedup probe that counts
|
||||
# open issues sharing the same title + automated label. Without this,
|
||||
# back-to-back failing runs (manual reruns, cron + workflow_dispatch)
|
||||
# farm one fresh issue per run instead of letting the open one
|
||||
# accumulate context.
|
||||
- name: Check for existing RPC Mismatch issue
|
||||
if: steps.health.outputs.exit_code == '1'
|
||||
id: dup_mismatch
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
if ! count=$(gh issue list --repo "$GITHUB_REPOSITORY" \
|
||||
--state open --label automated --label rpc-breakage \
|
||||
--search 'in:title "RPC ID Mismatch Detected"' \
|
||||
--json number --jq 'length'); then
|
||||
echo "::warning::RPC mismatch dedup probe failed; allowing issue creation"
|
||||
count=0
|
||||
fi
|
||||
echo "open=${count}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Create Issue on RPC Mismatch
|
||||
if: |
|
||||
steps.health.outputs.exit_code == '1'
|
||||
&& steps.dup_mismatch.outputs.open == '0'
|
||||
uses: peter-evans/create-issue-from-file@fca9117c27cdc29c6c4db3b86c48e4115a786710 # @ v6.0.0
|
||||
with:
|
||||
title: "RPC ID Mismatch Detected"
|
||||
content-filepath: health-report.txt
|
||||
labels: bug, rpc-breakage, automated
|
||||
|
||||
- name: Check for existing Auth Failure issue
|
||||
if: steps.health.outputs.exit_code == '2'
|
||||
id: dup_auth
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
if ! count=$(gh issue list --repo "$GITHUB_REPOSITORY" \
|
||||
--state open --label automated \
|
||||
--search 'in:title "RPC Health Check: Authentication Failure"' \
|
||||
--json number --jq 'length'); then
|
||||
echo "::warning::Auth failure dedup probe failed; allowing issue creation"
|
||||
count=0
|
||||
fi
|
||||
echo "open=${count}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Create Issue on Auth Failure
|
||||
if: |
|
||||
steps.health.outputs.exit_code == '2'
|
||||
&& steps.dup_auth.outputs.open == '0'
|
||||
uses: peter-evans/create-issue-from-file@fca9117c27cdc29c6c4db3b86c48e4115a786710 # @ v6.0.0
|
||||
with:
|
||||
title: "RPC Health Check: Authentication Failure"
|
||||
content-filepath: health-report.txt
|
||||
labels: bug, automated
|
||||
|
||||
- name: Check for existing non-transient ERROR issue
|
||||
if: steps.health.outputs.exit_code == '3'
|
||||
id: dup_error
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
if ! count=$(gh issue list --repo "$GITHUB_REPOSITORY" \
|
||||
--state open --label automated --label rpc-error \
|
||||
--search 'in:title "RPC Health Check: Non-transient ERROR detected"' \
|
||||
--json number --jq 'length'); then
|
||||
echo "::warning::Non-transient ERROR dedup probe failed; allowing issue creation"
|
||||
count=0
|
||||
fi
|
||||
echo "open=${count}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Extract failing methods for ERROR issue
|
||||
if: |
|
||||
steps.health.outputs.exit_code == '3'
|
||||
&& steps.dup_error.outputs.open == '0'
|
||||
id: error_details
|
||||
shell: bash
|
||||
run: |
|
||||
# Pull the affected-methods list out of the RESULT line emitted by the script.
|
||||
affected=$(grep -m1 "non-transient ERROR detected in methods:" health-report.txt \
|
||||
| sed -E 's/.*methods: //' || true)
|
||||
if [ -z "$affected" ]; then
|
||||
affected="(see report)"
|
||||
fi
|
||||
{
|
||||
echo "## Non-transient ERROR detected"
|
||||
echo ""
|
||||
echo "**Affected methods:** ${affected}"
|
||||
echo ""
|
||||
echo "**Commit:** ${GITHUB_SHA}"
|
||||
echo ""
|
||||
echo "Full report attached below (rate-limit / \`RESOURCE_EXHAUSTED\` errors"
|
||||
echo "are filtered out — anything listed here is a real failure that needs"
|
||||
echo "investigation: timeouts, parse failures, unexpected HTTP errors)."
|
||||
echo ""
|
||||
echo "---"
|
||||
echo ""
|
||||
cat health-report.txt
|
||||
} > error-issue-body.md
|
||||
|
||||
- name: Create Issue on Non-transient ERROR
|
||||
if: |
|
||||
steps.health.outputs.exit_code == '3'
|
||||
&& steps.dup_error.outputs.open == '0'
|
||||
uses: peter-evans/create-issue-from-file@fca9117c27cdc29c6c4db3b86c48e4115a786710 # @ v6.0.0
|
||||
with:
|
||||
title: "RPC Health Check: Non-transient ERROR detected"
|
||||
content-filepath: error-issue-body.md
|
||||
labels: rpc-error, bug, automated
|
||||
|
||||
# Exit code 4 = the ``sqTeoe`` cohort tripwire flipped (GetArtifactCustomization
|
||||
# Choices returned non-null). Our account migrated to the new studio-customization
|
||||
# surface; the VideoStyle / format codes in rpc/types.py must be re-captured.
|
||||
# GATED-null is the expected steady state and never reaches this branch.
|
||||
- name: Check for existing cohort-flip issue
|
||||
if: steps.health.outputs.exit_code == '4'
|
||||
id: dup_cohort
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
if ! count=$(gh issue list --repo "$GITHUB_REPOSITORY" \
|
||||
--state open --label automated --label rpc-breakage \
|
||||
--search 'in:title "Studio customization cohort flipped"' \
|
||||
--json number --jq 'length'); then
|
||||
echo "::warning::Cohort-flip dedup probe failed; allowing issue creation"
|
||||
count=0
|
||||
fi
|
||||
echo "open=${count}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Create Issue on cohort flip
|
||||
if: |
|
||||
steps.health.outputs.exit_code == '4'
|
||||
&& steps.dup_cohort.outputs.open == '0'
|
||||
uses: peter-evans/create-issue-from-file@fca9117c27cdc29c6c4db3b86c48e4115a786710 # @ v6.0.0
|
||||
with:
|
||||
title: "Studio customization cohort flipped — re-capture VideoStyle codes"
|
||||
content-filepath: health-report.txt
|
||||
labels: rpc-breakage, automated
|
||||
|
||||
- name: Upload Report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: rpc-health-report
|
||||
path: health-report.txt
|
||||
retention-days: 30
|
||||
|
||||
- name: Fail if health check failed
|
||||
if: steps.health.outcome == 'failure'
|
||||
run: |
|
||||
echo "RPC Health Check failed (exit code: ${{ steps.health.outputs.exit_code }}). See report above."
|
||||
exit 1
|
||||
@@ -0,0 +1,316 @@
|
||||
name: Test
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
custom_branch:
|
||||
description: 'Branch/ref to test (leave empty to use the triggering ref). Use this to run the full gate against a release branch that conflicts with main and so cannot run via pull_request.'
|
||||
required: false
|
||||
default: ''
|
||||
|
||||
concurrency:
|
||||
# Include the dispatch input so a manual run against a release branch gets its
|
||||
# own group instead of sharing (and cancelling) the push/PR run on the
|
||||
# triggering ref. Empty for push/pull_request, so their behavior is unchanged.
|
||||
group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.custom_branch }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
quality:
|
||||
name: Code Quality
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
# Honor the workflow_dispatch custom_branch input; falls back to the
|
||||
# triggering ref for push/pull_request (input is empty there).
|
||||
ref: ${{ inputs.custom_branch || github.ref }}
|
||||
fetch-depth: 0
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
cache: 'pip'
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
enable-cache: true
|
||||
|
||||
- name: Install dependencies
|
||||
shell: bash
|
||||
run: |
|
||||
# Retry to ride out transient registry/network blips during dep
|
||||
# resolution: a single PyPI/GitHub hiccup was hard-failing the leg.
|
||||
# `uv sync` has no internal retry across the full resolve+download.
|
||||
# A real lockfile problem fails all 3 attempts identically, so this
|
||||
# only absorbs flakes — it never masks a genuine resolution error.
|
||||
attempt=1; max=3
|
||||
until uv sync --frozen --extra browser --extra dev --extra markdown; do
|
||||
if [ "$attempt" -ge "$max" ]; then
|
||||
echo "::error::uv sync failed after $max attempts" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "::warning::uv sync attempt $attempt failed; retrying in $((attempt * 15))s" >&2
|
||||
sleep $((attempt * 15))
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
- name: Assert core imports without optional adapter extras
|
||||
# This job uses the canonical lean install (browser+dev+markdown, no
|
||||
# `mcp`/`server`). Since the test matrix now installs those extras, this
|
||||
# is the remaining per-PR guard that the core package still imports
|
||||
# without fastmcp/fastapi present — a stray top-level `import fastmcp` in
|
||||
# non-adapter code would ImportError here, catching the regression before
|
||||
# it reaches the release-time smoke test in publish.yml.
|
||||
#
|
||||
# `notebooklm_cli` (not `cli.grouped`) is imported because it is the
|
||||
# console-script entrypoint that builds the full command tree via
|
||||
# `cli.add_command(...)`, including `cli/mcp_cmd.py` — the most likely
|
||||
# place a stray `import fastmcp` would land. `cli.grouped` only defines
|
||||
# the Click Group subclass and traverses none of the command modules.
|
||||
# `client` + `artifacts` cover the primary public Python-API surface
|
||||
# (`from notebooklm import NotebookLMClient`). Since the lean install has
|
||||
# neither fastmcp nor fastapi, a stray top-level import of either from any
|
||||
# core/`_app` module reachable here fails the step. The `notebooklm-server`
|
||||
# adapter is intentionally NOT imported: it legitimately requires fastapi
|
||||
# (so it can't load on the lean install) and is exercised by the `test`
|
||||
# matrix instead — `server` is not wired into the CLI command tree.
|
||||
run: uv run python -c "import notebooklm, notebooklm.notebooklm_cli, notebooklm.client, notebooklm.artifacts"
|
||||
|
||||
- name: Run pre-commit checks
|
||||
run: uv run pre-commit run --all-files
|
||||
|
||||
- name: Run type checking
|
||||
run: uv run mypy src/notebooklm --ignore-missing-imports
|
||||
|
||||
- name: Assert workflow permissions are scoped
|
||||
run: uv run python scripts/check_workflow_permissions.py
|
||||
|
||||
- name: Assert workflow secret-bearing jobs are gated
|
||||
# Companion to the permissions check: prevents a new
|
||||
# ``${{ secrets.NAME }}`` reference from landing without picking up
|
||||
# a job-level ``environment:`` declaration or a step-level
|
||||
# ``is_standard`` guard. See docs/development.md →
|
||||
# "Workflow secret gates".
|
||||
run: uv run python scripts/check_workflow_secret_gates.py
|
||||
|
||||
- name: Assert third-party actions in privileged workflows are SHA-pinned
|
||||
# Supply-chain gate: every third-party action (non ``actions/*``) in
|
||||
# publish / testpypi-publish / claude / rpc-health / nightly /
|
||||
# verify-package must use a 40-char commit SHA, not a floating
|
||||
# ``@v1`` / ``@release/v1`` / branch ref. Dependabot bumps the
|
||||
# SHAs weekly via the grouped ``actions`` updates in
|
||||
# ``.github/dependabot.yml``.
|
||||
run: uv run python scripts/check_action_pinning.py
|
||||
|
||||
- name: Assert no deprecation targets the shipping version
|
||||
# Release gate: a DeprecationWarning must never name the version in
|
||||
# pyproject.toml as its removal target. Lapsed shims are allowlisted
|
||||
# inside the script with a tracking issue.
|
||||
run: uv run python scripts/check_deprecation_targets.py
|
||||
|
||||
- name: Assert coverage thresholds match
|
||||
run: uv run python scripts/check_coverage_thresholds.py
|
||||
|
||||
- name: Assert CI install matches CONTRIBUTING.md
|
||||
run: uv run python scripts/check_ci_install_parity.py
|
||||
|
||||
- name: Assert repo-structure map (docs/architecture.md) is fresh
|
||||
run: uv run python scripts/check_claude_md_freshness.py
|
||||
|
||||
- name: Assert doc module references are fresh
|
||||
# Companion to the CLAUDE.md freshness gate: guards the rest of the docs
|
||||
# against stale module paths. Every relative link into src/notebooklm/
|
||||
# must resolve, and every inline module ref in the live docs must point at
|
||||
# a real module (rare historical mentions are allowlisted, shrink-only).
|
||||
run: uv run python scripts/check_docs_module_refs.py
|
||||
|
||||
- name: Audit public API compatibility
|
||||
# ``--check-stale`` also fails when an allowlist entry matches no break
|
||||
# against the baseline (it is already in the baseline). This forces the
|
||||
# allowlist to be pruned at each release boundary instead of silently
|
||||
# accumulating cruft. See docs/releasing.md → prune-allowlist-at-release.
|
||||
run: uv run python scripts/audit_public_api_compat.py --check-stale
|
||||
|
||||
- name: Assert per-method RPC coverage
|
||||
# Static check: each RPCMethod member must have at least one test
|
||||
# reference AND at least one cassette body containing its RPC id.
|
||||
# Pre-existing gaps are grandfathered via PREEXISTING_GAPS inside the
|
||||
# script — that set is a one-way ratchet and must not grow.
|
||||
run: uv run python tests/scripts/check_method_coverage.py
|
||||
|
||||
- name: Verify e2e test fixtures
|
||||
run: uv run pytest tests/e2e --collect-only -q
|
||||
|
||||
test:
|
||||
name: Test (${{ matrix.os }}, Python ${{ matrix.python-version }})
|
||||
runs-on: ${{ matrix.os }}
|
||||
needs: quality
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [ubuntu-latest, macos-latest, windows-latest]
|
||||
python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
|
||||
env:
|
||||
# Pin Playwright's browser install dir to a single workspace-relative
|
||||
# path on every OS. The previous dual-path cache stanza (``~/.cache``
|
||||
# + ``~/AppData/Local``) intermittently failed the cache restore on
|
||||
# Windows with ``actions/cache@v5`` — the missing Linux path tripped
|
||||
# the post-restore validation on a cache hit. Pinning the path here
|
||||
# lets the cache stanza below carry exactly one entry on every OS.
|
||||
PLAYWRIGHT_BROWSERS_PATH: ${{ github.workspace }}/.playwright-browsers
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
# Honor the workflow_dispatch custom_branch input; falls back to the
|
||||
# triggering ref for push/pull_request (input is empty there).
|
||||
ref: ${{ inputs.custom_branch || github.ref }}
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
cache: 'pip'
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
enable-cache: true
|
||||
|
||||
- name: Install dependencies
|
||||
shell: bash
|
||||
run: |
|
||||
# Retry to ride out transient registry/network blips during dep
|
||||
# resolution: a single PyPI/GitHub hiccup on one matrix entry was
|
||||
# hard-failing the whole leg while the other 14 passed. `uv sync`
|
||||
# has no internal retry across the full resolve+download. A real
|
||||
# lockfile problem fails all 3 attempts identically, so this only
|
||||
# absorbs flakes — it never masks a genuine resolution error.
|
||||
#
|
||||
# `mcp` + `server` + `impersonate` are installed here (on top of the
|
||||
# canonical browser+dev+markdown set) so the MCP, server, and curl_cffi
|
||||
# suites (`tests/unit/mcp`, `tests/integration/mcp_vcr`, `tests/server`,
|
||||
# `tests/unit/test_curl_cffi_transport_poc.py`) run in the main matrix
|
||||
# instead of being importorskip'd — that gives
|
||||
# `src/notebooklm/{mcp,server}/**` and `_curl_cffi_transport.py` real
|
||||
# coverage in coverage.json (no 0% blind spot) across the full
|
||||
# 3.10–3.14 × 3-OS matrix, which also proves curl_cffi's native wheels
|
||||
# resolve on every OS/Python. `cookies` is deliberately excluded:
|
||||
# rookiepy fails on Python 3.13/3.14 (see check_ci_install_parity.py).
|
||||
#
|
||||
# Tradeoff: the deleted MCP/server jobs each enforced a *scoped*
|
||||
# `--cov=src/notebooklm/{mcp,server} --cov-fail-under=90`. mcp/server (and
|
||||
# the curl_cffi adapter) are now covered only by the global aggregate gate
|
||||
# below — no per-subpackage floor backfills them, so a subpackage could
|
||||
# regress within the aggregate's headroom. Add entries to
|
||||
# `[tool.notebooklm.per_file_coverage_floors]` if that backstop is
|
||||
# wanted back.
|
||||
attempt=1; max=3
|
||||
until uv sync --frozen --extra browser --extra dev --extra markdown --extra mcp --extra server --extra impersonate; do
|
||||
if [ "$attempt" -ge "$max" ]; then
|
||||
echo "::error::uv sync failed after $max attempts" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "::warning::uv sync attempt $attempt failed; retrying in $((attempt * 15))s" >&2
|
||||
sleep $((attempt * 15))
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
- name: Assert curl_cffi imports (native wheel; no silent skip)
|
||||
# The matrix installs --extra impersonate, so curl_cffi MUST import on every
|
||||
# OS/Python. It's a native wheel: if it installs but fails to import, the
|
||||
# importorskip'd curl_cffi suites would silently skip in the coverage run
|
||||
# rather than fail. This makes a broken native import a hard error.
|
||||
shell: bash
|
||||
run: uv run python -c "import curl_cffi"
|
||||
|
||||
- name: Get Playwright version
|
||||
id: playwright-version
|
||||
shell: bash
|
||||
run: |
|
||||
echo "version=$(uv pip show playwright | grep '^Version:' | cut -d' ' -f2)" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Cache Playwright browsers
|
||||
uses: actions/cache@v6
|
||||
# Tolerate transient cache-backend failures: a miss already falls
|
||||
# through to the install step below, and a backend hiccup should be
|
||||
# treated the same. Without this, a 29s backend flake on a single
|
||||
# matrix entry hard-fails the job and skips the entire test run.
|
||||
continue-on-error: true
|
||||
with:
|
||||
path: ${{ github.workspace }}/.playwright-browsers
|
||||
# ``-ws`` suffix invalidates archives keyed to the prior dual-path
|
||||
# stanza so first restores after this change land on the new path.
|
||||
key: playwright-${{ matrix.os }}-${{ steps.playwright-version.outputs.version }}-ws
|
||||
|
||||
- name: Install Playwright browsers
|
||||
run: uv run playwright install chromium
|
||||
|
||||
- name: Install Playwright system dependencies (Linux)
|
||||
if: runner.os == 'Linux'
|
||||
shell: bash
|
||||
run: |
|
||||
if ! timeout 180s uv run playwright install-deps chromium; then
|
||||
echo "::warning::Playwright system dependency installation timed out or failed."
|
||||
echo "::warning::Continuing so the non-browser test matrix can run."
|
||||
fi
|
||||
|
||||
- name: Assert cassettes are sanitized
|
||||
# Runs on all matrix entries (ubuntu, macos, windows × 5 Python versions).
|
||||
# An earlier bash-only gate was replaced by a portable Python invocation
|
||||
# so the check runs uniformly across the cross-platform test matrix.
|
||||
#
|
||||
# ``--strict`` flips the repair-allowlist from "best-effort suppressor"
|
||||
# to "must be empty" — the phase-2 cassette cleanup is done, so any
|
||||
# entry sneaking back in is a CI failure (P1-5).
|
||||
# ``--recursive`` extends the scan from ``tests/cassettes/*.yaml`` to
|
||||
# ``tests/cassettes/**/*.yaml`` so a recorder can't smuggle a leak
|
||||
# into a nested folder like ``tests/cassettes/gzip_coverage/`` (P1-5).
|
||||
run: uv run python tests/scripts/check_cassettes_clean.py --strict --recursive
|
||||
|
||||
- name: Check fixtures for credential leaks
|
||||
# The cassette guard above is scoped to ``tests/cassettes/*.yaml``. Golden
|
||||
# RPC fixtures live under ``tests/fixtures/`` as ``.json`` (and one
|
||||
# captured ``.html`` page), which embed the same WIZ_global_data shapes —
|
||||
# a Google API key smuggled into a golden HTML fixture would otherwise
|
||||
# slip past CI entirely (the GET_INTERACTIVE_HTML.json class). ``--secrets
|
||||
# -only`` scans .json/.html/.yaml for high-severity credential shapes
|
||||
# (Google auth tokens + API keys) without tripping on the intentional
|
||||
# placeholder content (``"Scrubbed ..."`` names, test emails) that fills
|
||||
# those fixtures.
|
||||
run: uv run python tests/scripts/check_cassettes_clean.py --secrets-only --recursive tests/fixtures
|
||||
|
||||
- name: Run tests with coverage
|
||||
# ``-n auto --dist loadgroup`` parallelizes the ~11k-test suite across the
|
||||
# runner's cores (pytest-xdist is already a dev dep). ``loadgroup`` honors
|
||||
# ``@pytest.mark.xdist_group`` — the isolation escape hatch for tests touching
|
||||
# process-global state (tests/integration/concurrency/conftest.py); unmarked
|
||||
# tests distribute like ``load``. The suite is parallel-safe via per-test
|
||||
# ``NOTEBOOKLM_HOME`` tmp isolation, and pytest-cov merges per-worker coverage
|
||||
# automatically, so the coverage.json + per-file floors below are unaffected.
|
||||
#
|
||||
# Emit coverage.json so the per-file floor check below can read it without
|
||||
# re-running the suite. ``term-missing`` is kept so build logs still show the
|
||||
# missing-lines summary.
|
||||
run: uv run pytest -n auto --dist loadgroup --cov=src/notebooklm --cov-report=term-missing --cov-report=json:coverage.json --cov-fail-under=90
|
||||
|
||||
- name: Assert per-file coverage floors
|
||||
# Linux-only because ``coverage.json`` on Windows records backslash
|
||||
# paths (e.g. ``src\notebooklm\cli\doctor.py``) that won't match the
|
||||
# forward-slash keys in ``[tool.notebooklm.per_file_coverage_floors]``
|
||||
# in pyproject.toml. macOS uses forward slashes and would also work,
|
||||
# but a single OS is enough — the floors are platform-independent.
|
||||
if: runner.os == 'Linux'
|
||||
run: uv run python scripts/check_coverage_thresholds.py --coverage-json coverage.json
|
||||
@@ -0,0 +1,129 @@
|
||||
name: Publish to TestPyPI
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
build-and-test:
|
||||
name: Build wheel and run release smoke
|
||||
runs-on: ubuntu-latest
|
||||
# Smoke install pulls `[browser,dev,markdown]` extras (pytest, playwright,
|
||||
# mypy, ruff, markdownify, transitive deps). A compromised dep here is a
|
||||
# supply-chain concern but cannot mint a Trusted Publishing OIDC token,
|
||||
# because this job is not granted `id-token: write`. The publish job
|
||||
# below carries that scope and runs nothing except the trusted PyPA
|
||||
# action. See issue #820 for the threat model.
|
||||
permissions:
|
||||
contents: read
|
||||
outputs:
|
||||
version: ${{ steps.version.outputs.version }}
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Get version from pyproject.toml
|
||||
id: version
|
||||
run: |
|
||||
VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml', 'rb'))['project']['version'])")
|
||||
echo "version=$VERSION" >> $GITHUB_OUTPUT
|
||||
echo "Publishing version: $VERSION"
|
||||
|
||||
- name: Install build tools
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install build==1.5.0
|
||||
|
||||
- name: Build package
|
||||
run: python -m build
|
||||
|
||||
- name: Upload distribution artifacts
|
||||
# Capture pristine wheel + sdist BEFORE any third-party code (smoke
|
||||
# install, playwright, pytest) runs against `dist/`. If we uploaded
|
||||
# after the smoke install, a compromised dev/test dep could mutate
|
||||
# `dist/` post-build, and the publish job would upload tampered bytes
|
||||
# — attested with the project's trusted OIDC identity. Uploading here
|
||||
# snapshots the build output before any attacker-controlled code has
|
||||
# touched the runner. Smoke install/test below still runs against
|
||||
# `dist/` for behavior validation but can no longer influence what
|
||||
# the publish job receives.
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # @ v7
|
||||
with:
|
||||
name: notebooklm-py-dist
|
||||
path: dist/
|
||||
if-no-files-found: error
|
||||
# One-day retention is intentional: publish consumes this artifact
|
||||
# immediately. See issue #829 for the recovery trade-off.
|
||||
retention-days: 1
|
||||
|
||||
- name: Install built wheel + canonical contributor extras in a clean venv
|
||||
# Canonical contributor extras (`[browser,dev,markdown]`, equivalent to
|
||||
# `[all]`) match the install path documented in `docs/installation.md`,
|
||||
# so the release smoke exercises the same surface area users will see.
|
||||
# Plain `[dev]` skipped playwright + markdownify, masking packaging bugs
|
||||
# in `[browser]`/`[markdown]` until users hit them post-release.
|
||||
run: |
|
||||
python -m venv smoke-venv
|
||||
WHEEL=$(ls dist/notebooklm_py-*.whl)
|
||||
smoke-venv/bin/pip install "${WHEEL}[browser,dev,markdown]"
|
||||
smoke-venv/bin/python -c "import notebooklm; print(notebooklm.__version__)"
|
||||
|
||||
- name: Install Playwright browser for smoke
|
||||
# `[browser]` only installs the playwright Python package; the chromium
|
||||
# binary + Linux system libs are needed for the unit-test smoke
|
||||
# (tests/unit/test_windows_compatibility.py drives sync_playwright()).
|
||||
run: |
|
||||
smoke-venv/bin/playwright install chromium
|
||||
smoke-venv/bin/playwright install-deps chromium
|
||||
|
||||
- name: Run unit tests against wheel
|
||||
run: smoke-venv/bin/pytest tests/unit -q --no-cov -x
|
||||
|
||||
publish:
|
||||
name: Publish to TestPyPI
|
||||
needs: build-and-test
|
||||
runs-on: ubuntu-latest
|
||||
environment: testpypi
|
||||
# Minimum-scope publish job: only the trusted PyPA action runs here. No
|
||||
# `actions/checkout`, no Python install, no third-party dependencies in
|
||||
# the runner environment — so the OIDC token request that `id-token:
|
||||
# write` enables has no co-tenant code that could exfiltrate it.
|
||||
permissions:
|
||||
id-token: write
|
||||
contents: read
|
||||
|
||||
steps:
|
||||
- name: Download distribution artifacts
|
||||
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # @ v8.0.1
|
||||
with:
|
||||
name: notebooklm-py-dist
|
||||
path: dist/
|
||||
|
||||
- name: Upload to TestPyPI
|
||||
uses: pypa/gh-action-pypi-publish@cef221092ed1bacb1cc03d23a2d87d1d172e277b # @ v1.14.0
|
||||
with:
|
||||
repository-url: https://test.pypi.org/legacy/
|
||||
# PEP 740 attestations: pypa/gh-action-pypi-publish generates an
|
||||
# in-toto attestation for the uploaded artifacts and publishes it
|
||||
# via PyPI's Trusted Publishing flow (OIDC, no API token).
|
||||
attestations: true
|
||||
|
||||
- name: Summary
|
||||
run: |
|
||||
echo "## Published to TestPyPI" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Version:** ${{ needs.build-and-test.outputs.version }}" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Package:** https://test.pypi.org/project/notebooklm-py/${{ needs.build-and-test.outputs.version }}/" >> $GITHUB_STEP_SUMMARY
|
||||
echo "" >> $GITHUB_STEP_SUMMARY
|
||||
echo "**Next Step:** Run the **Verify Package** workflow with source=testpypi" >> $GITHUB_STEP_SUMMARY
|
||||
@@ -0,0 +1,158 @@
|
||||
name: Verify Generated Artifacts
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Run at 8 AM UTC daily (2 hours after nightly e2e tests)
|
||||
- cron: '0 8 * * *'
|
||||
workflow_dispatch: # Allow manual trigger
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
verify:
|
||||
name: Verify Artifacts
|
||||
runs-on: ubuntu-latest
|
||||
if: github.repository == 'teng-lin/notebooklm-py'
|
||||
# Secret-bearing job. ``NOTEBOOKLM_AUTH_JSON`` and friends live only in
|
||||
# the ``protected-readonly`` GitHub Environment (issue #1009), so the
|
||||
# env binding is unconditional — every trigger sees the same secret
|
||||
# values. Add a ``required reviewers`` rule on the environment if you
|
||||
# want to block workflow_dispatch behind manual approval; scheduled
|
||||
# runs would queue too, so the env carries no protection rules today.
|
||||
# See docs/development.md → "Workflow secret gates".
|
||||
environment: protected-readonly
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
cache: 'pip'
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # @ v7.0.0
|
||||
|
||||
- name: Install dependencies
|
||||
# `uv sync --frozen` reproduces the lockfile-pinned dep tree. The
|
||||
# inline verification script below only touches the public client API
|
||||
# (no playwright / pytest / markdownify), so no extras are needed.
|
||||
run: uv sync --frozen
|
||||
|
||||
# Fail-fast preflight. Without this, ``NotebookLMClient.from_storage()``
|
||||
# below would die with a confusing "no storage" error when the secret
|
||||
# resolves empty (issue #1009).
|
||||
- name: Verify auth secret is present
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
NOTEBOOKLM_GENERATION_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_GENERATION_NOTEBOOK_ID }}
|
||||
shell: bash
|
||||
run: |
|
||||
if [ -z "${NOTEBOOKLM_AUTH_JSON:-}" ]; then
|
||||
echo "::error::NOTEBOOKLM_AUTH_JSON resolved to empty. Env binding or secret config is broken — see docs/development.md → Workflow secret gates."
|
||||
exit 1
|
||||
fi
|
||||
if [ -z "${NOTEBOOKLM_GENERATION_NOTEBOOK_ID:-}" ]; then
|
||||
echo "::error::NOTEBOOKLM_GENERATION_NOTEBOOK_ID resolved to empty."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Verify artifacts exist
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
NOTEBOOKLM_GENERATION_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_GENERATION_NOTEBOOK_ID }}
|
||||
run: |
|
||||
uv run python -c "
|
||||
import asyncio
|
||||
import os
|
||||
import sys
|
||||
from notebooklm import NotebookLMClient
|
||||
|
||||
# Type ID to display name mapping
|
||||
TYPE_NAMES = {
|
||||
1: 'Audio',
|
||||
2: 'Report', # Study Guide, Briefing Doc, Blog Post
|
||||
3: 'Video',
|
||||
4: 'Quiz/Flashcards',
|
||||
5: 'Mind Map',
|
||||
7: 'Infographic',
|
||||
8: 'Slide Deck',
|
||||
9: 'Data Table',
|
||||
}
|
||||
|
||||
# Expected type IDs from generation tests
|
||||
# Note: Type 2 is Reports (study guide), Type 4 is Quiz+Flashcards
|
||||
EXPECTED_TYPES = {1, 2, 3, 4, 5, 7, 8, 9}
|
||||
|
||||
async def verify():
|
||||
async with await NotebookLMClient.from_storage() as client:
|
||||
nb_id = os.environ['NOTEBOOKLM_GENERATION_NOTEBOOK_ID']
|
||||
print(f'Checking notebook: {nb_id}')
|
||||
|
||||
# List artifacts
|
||||
artifacts = await client.artifacts.list(nb_id)
|
||||
print(f'\nTotal artifacts: {len(artifacts)}')
|
||||
|
||||
# Group by type and status
|
||||
by_type = {}
|
||||
for a in artifacts:
|
||||
key = a._artifact_type
|
||||
if key not in by_type:
|
||||
by_type[key] = []
|
||||
by_type[key].append(a)
|
||||
|
||||
print('\nArtifacts by type:')
|
||||
for t in sorted(by_type.keys()):
|
||||
items = by_type[t]
|
||||
type_name = TYPE_NAMES.get(t, f'Unknown({t})')
|
||||
print(f' {type_name} (type {t}): {len(items)}')
|
||||
for a in items:
|
||||
status = a.status_str
|
||||
variant_info = f', variant={a._variant}' if a._variant else ''
|
||||
print(f' - {a.title} ({status}{variant_info})')
|
||||
|
||||
# Check expected types
|
||||
found = set(by_type.keys())
|
||||
missing = EXPECTED_TYPES - found
|
||||
|
||||
print(f'\nExpected types: {len(EXPECTED_TYPES)}')
|
||||
print(f'Found types: {len(found)}')
|
||||
|
||||
if missing:
|
||||
missing_names = [TYPE_NAMES.get(t, str(t)) for t in missing]
|
||||
print(f'\nWARNING: Missing artifact types: {missing_names}')
|
||||
|
||||
# Check for completed artifacts
|
||||
completed = sum(1 for a in artifacts if a.is_completed)
|
||||
processing = sum(1 for a in artifacts if a.is_processing)
|
||||
failed = sum(1 for a in artifacts if a.is_failed)
|
||||
|
||||
print(f'\nStatus summary:')
|
||||
print(f' Completed: {completed}')
|
||||
print(f' Processing: {processing}')
|
||||
print(f' Failed: {failed}')
|
||||
|
||||
# List notes
|
||||
notes = await client.notes.list(nb_id)
|
||||
print(f'\nTotal notes: {len(notes)}')
|
||||
for n in notes[:10]:
|
||||
print(f' - {n.title or \"(untitled)\"}')
|
||||
if len(notes) > 10:
|
||||
print(f' ... and {len(notes) - 10} more')
|
||||
|
||||
# Fail if too many failures or no artifacts at all
|
||||
if len(artifacts) == 0:
|
||||
print('\nERROR: No artifacts found!')
|
||||
sys.exit(1)
|
||||
if failed > len(artifacts) // 2:
|
||||
print(f'\nERROR: Too many failed artifacts ({failed}/{len(artifacts)})')
|
||||
sys.exit(1)
|
||||
|
||||
print('\nVerification complete!')
|
||||
|
||||
asyncio.run(verify())
|
||||
"
|
||||
@@ -0,0 +1,199 @@
|
||||
name: Verify Package
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
source:
|
||||
description: 'Package source'
|
||||
required: true
|
||||
default: 'testpypi'
|
||||
type: choice
|
||||
options:
|
||||
- testpypi
|
||||
- pypi
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
verify:
|
||||
name: Verify from ${{ inputs.source }}
|
||||
runs-on: ubuntu-latest
|
||||
# Secret-bearing job (steps below reference secrets.NOTEBOOKLM_AUTH_JSON
|
||||
# + secrets.NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID for the live E2E pass). The
|
||||
# workflow is workflow_dispatch-only, so every run is human-initiated;
|
||||
# this `protected-readonly` environment requires a maintainer to approve
|
||||
# the dispatch before any secret is exposed. Non-maintainer dispatches
|
||||
# block at the approval prompt rather than acquiring tokens.
|
||||
# See docs/development.md → "Workflow secret gates".
|
||||
environment: protected-readonly
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Set up Python 3.12
|
||||
uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.12"
|
||||
cache: 'pip'
|
||||
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # @ v7.0.0
|
||||
|
||||
- name: Get version from pyproject.toml
|
||||
id: version
|
||||
run: |
|
||||
VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml', 'rb'))['project']['version'])")
|
||||
echo "version=$VERSION" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Sync locked deps + non-cookies extras
|
||||
# Pull every runtime + tooling dep from `uv.lock` (the same lockfile
|
||||
# contributors install from) so the verify step exercises the full
|
||||
# non-cookies dep tree rather than whatever pip happens to resolve at
|
||||
# smoke time. `cookies` stays excluded because rookiepy has Python
|
||||
# 3.13+ install issues.
|
||||
# The published wheel itself is force-reinstalled in the next step.
|
||||
run: >
|
||||
uv sync --frozen
|
||||
--extra browser
|
||||
--extra dev
|
||||
--extra markdown
|
||||
--extra mcp
|
||||
--extra server
|
||||
|
||||
- name: Install published wheel from TestPyPI (--no-deps)
|
||||
if: inputs.source == 'testpypi'
|
||||
shell: bash
|
||||
run: |
|
||||
# --no-deps proves the wheel was actually uploaded to TestPyPI — the
|
||||
# previous `--extra-index-url https://pypi.org/simple/` fallback would
|
||||
# silently succeed by resolving an older published version from PyPI
|
||||
# when the TestPyPI upload itself was broken or missing.
|
||||
# --reinstall replaces the editable install left behind by `uv sync`
|
||||
# with the actual built wheel under test. --no-cache + --only-binary
|
||||
# ensure we test the freshly-uploaded wheel, never a cached sdist.
|
||||
#
|
||||
# `uv pip install --python .venv/bin/python` is load-bearing: a
|
||||
# `source .venv/bin/activate && pip install …` chain falls back to
|
||||
# the runner's system pip when uv has not seeded pip into `.venv`,
|
||||
# which then writes outside the venv and leaves the editable install
|
||||
# in place — masking a broken/missing TestPyPI upload.
|
||||
uv pip install --python .venv/bin/python \
|
||||
--no-deps --reinstall --no-cache --only-binary=:all: \
|
||||
--index-url https://test.pypi.org/simple/ \
|
||||
"notebooklm-py==${{ steps.version.outputs.version }}"
|
||||
|
||||
- name: Install published wheel from PyPI (--no-deps)
|
||||
if: inputs.source == 'pypi'
|
||||
shell: bash
|
||||
run: |
|
||||
# Symmetric with the TestPyPI step: --no-deps + --reinstall swaps the
|
||||
# locked editable install for the actual PyPI wheel without
|
||||
# disturbing the dep tree resolved from `uv.lock`. See the TestPyPI
|
||||
# step above for why `uv pip install --python .venv/bin/python` is
|
||||
# used instead of activating the venv and calling bare `pip`.
|
||||
uv pip install --python .venv/bin/python \
|
||||
--no-deps --reinstall --no-cache --only-binary=:all: \
|
||||
"notebooklm-py==${{ steps.version.outputs.version }}"
|
||||
|
||||
- name: Verify version
|
||||
shell: bash
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
INSTALLED=$(python -c "from notebooklm import __version__; print(__version__)")
|
||||
EXPECTED="${{ steps.version.outputs.version }}"
|
||||
if [ "$INSTALLED" != "$EXPECTED" ]; then
|
||||
echo "Version mismatch: installed=$INSTALLED expected=$EXPECTED"
|
||||
exit 1
|
||||
fi
|
||||
echo "Version verified: $INSTALLED"
|
||||
|
||||
- name: Verify CLI
|
||||
shell: bash
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
notebooklm --version
|
||||
notebooklm --help
|
||||
|
||||
- name: Verify imports
|
||||
shell: bash
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
python -c "from notebooklm import NotebookLMClient, Notebook, Source, Artifact"
|
||||
echo "Core imports verified"
|
||||
|
||||
- name: Install Playwright browsers
|
||||
shell: bash
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
# Test deps come from the non-cookies extras installed via
|
||||
# `uv sync --frozen` above. Just need to provision the Chromium browser
|
||||
# binary + Linux system libs.
|
||||
playwright install chromium
|
||||
playwright install-deps chromium
|
||||
|
||||
- name: Run unit tests
|
||||
shell: bash
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
pytest tests/unit -v
|
||||
|
||||
- name: Run integration tests
|
||||
shell: bash
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
pytest tests/integration -v
|
||||
|
||||
- name: Skip E2E tests (fork)
|
||||
if: github.repository != 'teng-lin/notebooklm-py'
|
||||
run: |
|
||||
echo "::warning::E2E tests skipped - secrets not available in fork repositories"
|
||||
echo "## E2E Tests Skipped" >> $GITHUB_STEP_SUMMARY
|
||||
echo "E2E tests were not run because this workflow is executing on a fork." >> $GITHUB_STEP_SUMMARY
|
||||
|
||||
- name: Run E2E tests
|
||||
id: e2e
|
||||
if: github.repository == 'teng-lin/notebooklm-py'
|
||||
# Don't fail the job here — the retry step below gets a 10-min cool-down
|
||||
# shot at any failures, and its exit code is what marks the job red.
|
||||
continue-on-error: true
|
||||
shell: bash
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID }}
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
if [ -z "$NOTEBOOKLM_AUTH_JSON" ]; then
|
||||
echo "::error::NOTEBOOKLM_AUTH_JSON secret is not configured"
|
||||
exit 1
|
||||
fi
|
||||
if [ -z "$NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID" ]; then
|
||||
echo "::error::NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID secret is not configured"
|
||||
exit 1
|
||||
fi
|
||||
# --reruns 1 --reruns-delay 30 catches sub-minute network blips;
|
||||
# longer chat-throttle recovery is handled by the cool-down retry below.
|
||||
pytest tests/e2e -m "not variants" --reruns 1 --reruns-delay 30 -v
|
||||
|
||||
- name: Retry failed E2E tests after 10-min cool-down
|
||||
# Tests still throttled after the cool-down become skips via the
|
||||
# _install_chat_rate_limit_skip fixture in tests/e2e/conftest.py.
|
||||
if: steps.e2e.outcome == 'failure'
|
||||
shell: bash
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID: ${{ secrets.NOTEBOOKLM_READ_ONLY_NOTEBOOK_ID }}
|
||||
run: |
|
||||
source .venv/bin/activate
|
||||
# Missing lastfailed after a failed e2e step means pytest crashed
|
||||
# before writing the cache (import error, OOM, etc.) — fail the job
|
||||
# rather than silently going green.
|
||||
if [ ! -f .pytest_cache/v/cache/lastfailed ]; then
|
||||
echo "::error::No lastfailed cache from previous step; pytest likely crashed before running tests."
|
||||
exit 1
|
||||
fi
|
||||
echo "Initial E2E run failed. Sleeping 600s before retrying failed tests."
|
||||
sleep 600
|
||||
uv run pytest tests/e2e --last-failed --last-failed-no-failures=none -s -v --tb=short
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*.class
|
||||
.venv/
|
||||
env/
|
||||
venv/
|
||||
.pytest_cache/
|
||||
.coverage
|
||||
coverage.json
|
||||
htmlcov/
|
||||
dist/
|
||||
build/
|
||||
*.egg-info/
|
||||
.DS_Store
|
||||
.env
|
||||
.notebooklm/
|
||||
captured_rpcs/
|
||||
.worktrees/
|
||||
.worktree/
|
||||
worktrees/
|
||||
.sisyphus/
|
||||
.scratch/
|
||||
.claude/
|
||||
|
||||
# Auto-generated by claude-mem plugin
|
||||
**/CLAUDE.md
|
||||
!/CLAUDE.md
|
||||
|
||||
# Investigation artifacts
|
||||
investigate*.py
|
||||
|
||||
# Playwright browser binaries when PLAYWRIGHT_BROWSERS_PATH is set to a
|
||||
# workspace-relative path (used by CI; harmless locally if it ever lands).
|
||||
.playwright-browsers/
|
||||
INVESTIGATION*.md
|
||||
investigation_output/
|
||||
downloads/
|
||||
|
||||
# VCR cassettes - committed after security review
|
||||
# Cassettes are scrubbed of sensitive data (cookies, tokens, user IDs, emails)
|
||||
# See tests/vcr_config.py for scrubbing patterns
|
||||
|
||||
# Local working notes (process scratch — not shipped)
|
||||
docs/scratch/
|
||||
|
||||
# Baked into the wheel by hatch_build.py; never committed.
|
||||
src/notebooklm/_commit.py
|
||||
@@ -0,0 +1,19 @@
|
||||
# Ruff is pinned once in pyproject.toml ([project.optional-dependencies].dev:
|
||||
# `ruff==…`).
|
||||
# These local hooks invoke that same interpreter's ruff via `uv run`, so the
|
||||
# pre-commit version can never drift from the version used by CI and locally.
|
||||
repos:
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: ruff
|
||||
name: ruff
|
||||
entry: uv run ruff check --force-exclude --fix
|
||||
language: system
|
||||
types_or: [python, pyi]
|
||||
require_serial: true
|
||||
- id: ruff-format
|
||||
name: ruff-format
|
||||
entry: uv run ruff format --force-exclude
|
||||
language: system
|
||||
types_or: [python, pyi]
|
||||
require_serial: true
|
||||
@@ -0,0 +1,44 @@
|
||||
# Repository Guidelines
|
||||
|
||||
**Status:** Active
|
||||
**Last Updated:** 2026-06-11
|
||||
|
||||
## Project Structure & Module Organization
|
||||
|
||||
`src/notebooklm/` contains the async client and typed APIs. Internal feature modules use `_` prefixes such as `_sources.py`, `_artifacts.py`, `_app/`, and `_runtime/`; `src/notebooklm/cli/` holds Click adapters, `src/notebooklm/mcp/` and `src/notebooklm/server/` hold the opt-in MCP and REST adapters, and `src/notebooklm/rpc/` handles protocol encoding and decoding. Tests are split by scope: `tests/unit/`, `tests/integration/`, `tests/server/`, and `tests/e2e/`. Recorded HTTP fixtures live in `tests/cassettes/`. Examples are in `examples/`, and diagnostics live in `scripts/`.
|
||||
|
||||
## Build, Test, and Development Commands
|
||||
|
||||
Canonical contributor install (full guide: [docs/installation.md](docs/installation.md)):
|
||||
|
||||
```bash
|
||||
uv sync --frozen --extra browser --extra dev --extra markdown
|
||||
source .venv/bin/activate
|
||||
uv run playwright install chromium
|
||||
uv run pytest
|
||||
uv run pytest -n auto --dist=worksteal # optional faster local run
|
||||
uv run ruff check .
|
||||
uv run ruff format .
|
||||
uv run mypy src/notebooklm
|
||||
uv run pre-commit run --all-files
|
||||
```
|
||||
|
||||
Run `uv run pytest tests/e2e -m readonly` only after `notebooklm login` and setting test notebook env vars.
|
||||
|
||||
## Coding Style & Naming Conventions
|
||||
|
||||
Target Python 3.10+, 4-space indentation, and double quotes. Ruff enforces formatting and import order with a 100-character line length. Keep module and test file names in `snake_case`; prefer descriptive Click command names that match existing groups such as `source`, `label`, `artifact`, and `research`. Preserve the internal/public split: `_*.py` and `_*/` for implementation, exported types in `src/notebooklm/__init__.py`.
|
||||
|
||||
## Testing Guidelines
|
||||
|
||||
Put pure logic in `tests/unit/`, REST adapter coverage in `tests/server/`, VCR-backed flows in `tests/integration/`, and authenticated NotebookLM coverage in `tests/e2e/`. Name tests `test_<behavior>.py` and record cassettes with `NOTEBOOKLM_VCR_RECORD=1 uv run pytest tests/integration/ -v` (the integration suite uses `vcrpy` throughout — there is no `test_vcr_*.py` glob). Coverage is expected to stay at or above the configured 90% threshold.
|
||||
|
||||
## Commit, PR, and Agent Notes
|
||||
|
||||
Follow the existing commit style: `feat(cli): ...`, `fix(cli): ...`, `refactor(test): ...`, `style: ...`. PRs should include a short summary, linked issue when relevant, and the commands run locally.
|
||||
|
||||
For Codex or other parallel agents:
|
||||
|
||||
- Prefer `--json` output and pass explicit notebook IDs instead of relying on `notebooklm use`.
|
||||
- Isolate concurrent runs with `NOTEBOOKLM_PROFILE=agent-<id>` so each agent gets its own context file under `~/.notebooklm/profiles/<name>/`. Fall back to `NOTEBOOKLM_HOME=/tmp/agent-<id>` only when separate home directories are required.
|
||||
- In headless environments where Playwright login is impractical, authenticate with `notebooklm login --browser-cookies <browser>` (requires `pip install "notebooklm-py[cookies]"`).
|
||||
+2109
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,84 @@
|
||||
# CLAUDE.md
|
||||
|
||||
Guidance for Claude Code working in this repo. Also follow the file/naming conventions in [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
## Project Overview
|
||||
|
||||
`notebooklm-py` is an unofficial **async** Python client for Google NotebookLM. It drives Google's internal `batchexecute` RPC protocol to automate notebooks, sources, AI querying, and studio artifacts (podcasts, videos, quizzes, …).
|
||||
|
||||
**Critical constraint:** the obfuscated RPC method IDs in `src/notebooklm/rpc/types.py` are undocumented and can break whenever Google changes them — the #1 breakage class.
|
||||
|
||||
## Development Commands
|
||||
|
||||
```bash
|
||||
# Canonical contributor install (respects uv.lock; full guide: docs/installation.md)
|
||||
uv sync --frozen --extra browser --extra dev --extra markdown
|
||||
source .venv/bin/activate
|
||||
uv run playwright install chromium
|
||||
|
||||
uv run pytest # all tests (e2e excluded by default)
|
||||
uv run pytest --cov # with coverage
|
||||
uv run pytest tests/e2e -m e2e # e2e (requires auth)
|
||||
uv run notebooklm --help # CLI
|
||||
```
|
||||
|
||||
## Before Pushing
|
||||
|
||||
The pre-commit hook runs ruff (format + lint) on staged files. Also run these manually — CI fails otherwise:
|
||||
|
||||
```bash
|
||||
uv run mypy src/notebooklm --ignore-missing-imports
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
`cli/` (Click) → `_app/` (transport-neutral business logic, reusable by MCP/HTTP adapters) → `client.py` + `_*.py` (client runtime) → `rpc/` (batchexecute encode/decode).
|
||||
|
||||
See **[docs/architecture.md](docs/architecture.md)** for the layered design, call flows, cross-cutting policies (loop affinity, idempotency, schema validation), the per-file index, and the full repository tree.
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
1. **RPC method IDs change** — re-capture network traffic and update `rpc/types.py`.
|
||||
2. **Position-sensitive nested params** — copy the shape from an existing implementation; source-id nesting varies (`[id]` / `[[id]]` / `[[[id]]]` / `[[[[id]]]]`).
|
||||
3. **CSRF tokens expire** — call `client.refresh_auth()` or re-run `notebooklm login`.
|
||||
4. **Rate limiting** — add delays between bulk operations.
|
||||
5. **Concurrency** — one `NotebookLMClient` is bound to its `open()`-time event loop: create one per thread, never reuse across event loops or `AuthTokens` tenants. See the [concurrency contract](docs/python-api.md#concurrency-contract).
|
||||
|
||||
## Usage
|
||||
|
||||
```python
|
||||
async with NotebookLMClient.from_storage() as client:
|
||||
notebooks = await client.notebooks.list()
|
||||
await client.sources.add_url(nb_id, url)
|
||||
answer = await client.chat.ask(nb_id, question)
|
||||
status = await client.artifacts.generate_audio(nb_id)
|
||||
```
|
||||
|
||||
CLI: top-level commands (`login`, `use`, `status`, `list`, `ask`) plus grouped subcommands (`source add`, `label list`, `artifact list`, `generate audio`, `download video`, `note create`, `mcp install <client>`, …). Full reference: [docs/cli-reference.md](docs/cli-reference.md).
|
||||
|
||||
An opt-in MCP server (`mcp` extra, console script `notebooklm-mcp`) exposes the same `_app/` business logic over the Model Context Protocol; `notebooklm mcp install <client>` wires it into Claude Desktop/Code, Cursor, or Windsurf, and `desktop-extension/` packages a one-click `.mcpb` bundle.
|
||||
|
||||
An opt-in single-tenant REST server (`server` extra, console script `notebooklm-server`) exposes guarded `/v1` FastAPI routes over the same `_app/` layer. It is experimental, loopback-bound by default, and requires `NOTEBOOKLM_SERVER_TOKEN`; see [docs/installation.md#rest-api-server](docs/installation.md#rest-api-server).
|
||||
|
||||
## Testing
|
||||
|
||||
Unit (`tests/unit/`, no network; includes `_app`, CLI, server, and guardrail tests) · integration (`tests/integration/`, VCR cassette replay) · e2e (`tests/e2e/`, real API, `@pytest.mark.e2e`). VCR cassettes match on `rpcids` + decoded body shape. Details: [docs/development.md](docs/development.md).
|
||||
|
||||
## Docs
|
||||
|
||||
`docs/`: installation · cli-reference · python-api · configuration · troubleshooting · development · architecture · mcp-guide · rpc-development · rpc-reference · stability · adr/.
|
||||
|
||||
## Pull Request Workflow (required)
|
||||
|
||||
After opening a PR, drive it to merge:
|
||||
|
||||
1. Poll `gh pr checks <PR>` until all pass; investigate and fix any failures.
|
||||
2. Address every review comment (especially `gemini-code-assist`): make the fix, push, then reply on the thread (`Addressed in <SHA>: …`). Unreplied threads block merge.
|
||||
3. Not done until all checks pass, all threads addressed, and `mergeStateStatus` is `CLEAN`.
|
||||
|
||||
Claude review is **not** automatic — comment `@claude review` on the PR to trigger the `.github/workflows/claude.yml` workflow. Treat `claude[bot]` as a first-class reviewer the merge gate **waits on**, alongside `gemini-code-assist` / `coderabbitai`:
|
||||
|
||||
- It posts inline review-thread comments **plus** a sticky summary comment ("**Claude finished … task**"). The action does **not** submit a formal GitHub review, so `claude[bot]` never appears in `gh pr view --json reviews` / `reviewDecision` and is **not** a required check — do not infer "claude reviewed" from those.
|
||||
- `gh pr checks` may show a `claude` entry as **skipping**: every comment (incl. other bots') fires `claude.yml`, and runs not from a `teng-lin` `@claude` comment correctly skip via the job `if:` gate. That skip is **not** the review run — find the real one with `gh run list --workflow=claude.yml --json event,conclusion` (look for the `success` run) or just read the comment below.
|
||||
- Before merging, confirm the review landed and address it. The two halves live on different endpoints: the sticky summary is an **issue** comment (`gh api /repos/<owner>/<repo>/issues/<PR>/comments`), and the inline findings are **pull-request review** comments on the diff (`gh api /repos/<owner>/<repo>/pulls/<PR>/comments`) — filter either with `--jq '.[]|select(.user.login=="claude[bot]").body'`. Resolve any inline `claude[bot]` threads like any other bot's.
|
||||
+252
@@ -0,0 +1,252 @@
|
||||
# Contributing to notebooklm-py
|
||||
|
||||
## For Human Contributors
|
||||
|
||||
### Getting Started
|
||||
|
||||
```bash
|
||||
# Canonical contributor install (respects uv.lock)
|
||||
uv sync --frozen --extra browser --extra dev --extra markdown
|
||||
source .venv/bin/activate
|
||||
uv run playwright install chromium
|
||||
pre-commit install
|
||||
```
|
||||
|
||||
**Run the full pre-commit suite** (matches what CI runs). IMPORTANT: use the
|
||||
broad `.` scope, not `src/ tests/` — the pre-commit hook in CI invokes
|
||||
ruff-format on the whole tree and is stricter than a narrow scope.
|
||||
|
||||
```bash
|
||||
uv run ruff format --check . && \
|
||||
uv run ruff check . && \
|
||||
uv run mypy src/notebooklm --ignore-missing-imports && \
|
||||
uv run pytest --cov=src/notebooklm --cov-report=term-missing --cov-fail-under=90
|
||||
```
|
||||
|
||||
**No uv?** Plain pip works as a fallback (won't enforce the lockfile, so you may resolve newer dep versions than CI):
|
||||
```bash
|
||||
python -m venv .venv && source .venv/bin/activate
|
||||
pip install -e ".[all]" # [all] = browser + dev + markdown + mcp + server (no cookies; see installation.md)
|
||||
playwright install chromium
|
||||
pre-commit install
|
||||
```
|
||||
|
||||
For full prerequisites, headless setup, optional extras (`[cookies]`, `[markdown]`, `[mcp]`, `[server]`), and platform notes, see [docs/installation.md#e-contributor](docs/installation.md#e-contributor).
|
||||
|
||||
> **Install-doc parity.** `docs/installation.md` is the canonical install guide; this file mirrors a small contributor-focused subset. Every fenced ``bash`` block in `installation.md` must EITHER appear verbatim in `CONTRIBUTING.md`, OR be marked with `<!-- not mirrored: <reason> -->` on the line directly before its opening fence. CI enforces this via `scripts/check_ci_install_parity.py` so a stale block can't drift in unnoticed. When you edit `installation.md`, decide on the spot whether the new content also belongs in this file.
|
||||
|
||||
The `browser` extra is part of the contributor install because the default unit
|
||||
suite imports and patches `playwright.sync_api`. The command
|
||||
`uv sync --frozen --extra dev` is only the test/lint toolchain; it is not enough
|
||||
for `uv run pytest`.
|
||||
|
||||
> **Architecture & testing context.** Once installed, read [docs/development.md](docs/development.md) for the layered RPC/Core/Client/CLI design, test-tree layout, and release workflow before touching `src/notebooklm/`.
|
||||
|
||||
### Code Quality
|
||||
|
||||
This project uses **ruff** for linting and formatting:
|
||||
|
||||
```bash
|
||||
# Check for lint issues
|
||||
ruff check .
|
||||
|
||||
# Auto-fix lint issues
|
||||
ruff check --fix .
|
||||
|
||||
# Check formatting
|
||||
ruff format --check .
|
||||
|
||||
# Apply formatting
|
||||
ruff format .
|
||||
```
|
||||
|
||||
**Pre-commit hooks** (included in the `[dev]` extra; install once after the canonical setup):
|
||||
```bash
|
||||
pre-commit install # one-time, after the canonical install
|
||||
pre-commit run --all-files # manual run on the whole tree (matches the CI lint gate)
|
||||
```
|
||||
|
||||
> **Caveat:** if `pre-commit install` errors with `Cowardly refusing to install hooks with core.hooksPath set`, your git is configured to use a custom hooks directory (common with Husky / nx / shared dev configs). Workaround: `git config --unset core.hooksPath` then re-run `pre-commit install`, or run `pre-commit run --all-files` manually before each commit. CI runs the same hook either way, so a clean local hook is convenience, not correctness.
|
||||
|
||||
> **CI parity.** The local pre-commit one-liner above matches the CI **lint gate** (`uv run pre-commit run --all-files` in `.github/workflows/test.yml`). CI additionally runs the full test matrix on multiple Python versions (3.10–3.14) and asserts a 90% coverage floor (`pytest --cov=src/notebooklm --cov-report=term-missing --cov-fail-under=90`). The lint+test failure modes are caught locally; the multi-Python-version drift is not — `uv run pytest --cov=src/notebooklm --cov-report=term-missing --cov-fail-under=90` here uses your local Python version only.
|
||||
|
||||
### Pull Request Process
|
||||
|
||||
1. Create a feature branch from `main`
|
||||
2. Make your changes with clear commit messages
|
||||
3. Ensure tests pass: `pytest`
|
||||
4. Ensure lint passes: `ruff check .`
|
||||
5. Ensure formatting: `ruff format --check .`
|
||||
6. Submit a PR with a description of changes
|
||||
|
||||
### Pull Request Quality Expectations
|
||||
|
||||
- **Reference an issue**: PRs should link to an existing issue or clearly describe the problem being solved. If no issue exists, open one first for discussion.
|
||||
- **AI-assisted contributions**: Welcome, but the submitter must review, understand, and test the code before submitting. PRs that appear to be unreviewed AI output will be closed.
|
||||
- **No duplicates**: Check existing open PRs before submitting. Duplicate PRs for the same issue will be closed in favor of the first or best submission.
|
||||
- **Accurate severity**: Claims of "critical" bugs must include evidence (stack trace, reproduction steps, affected users). Routine edge cases are not critical.
|
||||
- **Tested locally**: All PRs must include evidence of local testing. The PR template includes a checklist for this.
|
||||
|
||||
### Dependency upper bounds
|
||||
|
||||
Every runtime and `[project.optional-dependencies]` entry in `pyproject.toml` must have an upper bound — typically `<currentmajor + 1` (or `<currentminor + 1` for pre-1.0 packages like `httpx`). The bound protects downstream installs from a breaking new release that lands before we have time to test it.
|
||||
|
||||
When you bump a cap (e.g. moving `pytest>=8.0,<10` to `pytest>=8.0,<11`):
|
||||
|
||||
1. Run `uv lock --refresh` and `uv sync --frozen --extra browser --extra dev --extra markdown` locally.
|
||||
2. Run the full pre-commit one-liner above.
|
||||
3. Mention the upgrade rationale in the PR description.
|
||||
|
||||
The `dependency-audit` workflow (`.github/workflows/dependency-audit.yml`) runs `pip-audit --strict --require-hashes` against the locked env on every push to `main` and nightly. It is a **hard gate** (no `continue-on-error`): a CVE in the locked environment fails the workflow. New deps must pass `pip-audit` cleanly when introduced; pin a fixed version or record a tracked exception via `pip-audit`'s `--ignore-vuln` if no fix is yet available.
|
||||
|
||||
### Test tiers
|
||||
|
||||
The test suite is split into three tiers by network/auth dependency. Place new tests in the tier that matches their isolation profile — a tier-enforcement hook will eventually fail PRs that mis-tier a test.
|
||||
|
||||
| Tier | Location | What lives here | Network | Auth |
|
||||
|------|----------|-----------------|---------|------|
|
||||
| Unit | `tests/unit/` | Pure-Python tests + `pytest_httpx` (`httpx_mock`) request-level mocks. Encoder/decoder, dataclasses, helpers, CLI boundary, and httpx_mock-driven API tests. | None (mocked) | None |
|
||||
| Integration | `tests/integration/` | VCR cassette replay only — `@pytest.mark.vcr` / `notebooklm_vcr.use_cassette(...)` against recorded fixtures in `tests/cassettes/`. | None (replayed) | None |
|
||||
| E2E | `tests/e2e/` | Real NotebookLM API. Marked `@pytest.mark.e2e`; excluded from the default `pytest` run via `addopts = --ignore=tests/e2e`. | Real | Required (`notebooklm login`) |
|
||||
|
||||
Run a tier explicitly:
|
||||
|
||||
```bash
|
||||
uv run pytest tests/unit
|
||||
uv run pytest tests/integration
|
||||
uv run pytest tests/e2e -m e2e # requires auth
|
||||
```
|
||||
|
||||
#### Fast local loop (skip repo-wide audit checks)
|
||||
|
||||
A subset of unit tests are repo-wide audit / release-gate checks (cassette
|
||||
shape lint, public-surface scans, CI-script audits, doc-sync guards) that scan
|
||||
many files and add ~30–45s to the local `tests/unit tests/integration` loop.
|
||||
They're marked `@pytest.mark.repo_lint` so you can opt out while iterating:
|
||||
|
||||
```bash
|
||||
# Fast feedback loop — drops repo_lint audits (~40s savings).
|
||||
uv run pytest tests/unit tests/integration -m "not repo_lint"
|
||||
|
||||
# Run only the repo_lint audits (what you'd typically skip above).
|
||||
uv run pytest tests/unit tests/integration -m "repo_lint"
|
||||
```
|
||||
|
||||
Run the full suite (including `repo_lint`) before pushing — CI runs everything
|
||||
by default, so `repo_lint` failures still block merge. The default
|
||||
`uv run pytest` invocation does not filter the marker out.
|
||||
|
||||
Quick guidance:
|
||||
|
||||
- Reach for `httpx_mock` when you need to assert on outgoing request shape (headers, body, cookies, URL) or stub a small response — put the test under `tests/unit/`.
|
||||
- Reach for VCR when you want to replay a recorded server response end-to-end against the real RPC decoder — put the test under `tests/integration/` and record the cassette per `docs/rpc-development.md`.
|
||||
- Reach for E2E only when you need to validate a live round-trip against Google's servers — put the test under `tests/e2e/` and mark it `@pytest.mark.e2e`.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Rules for AI Agents
|
||||
|
||||
**IMPORTANT:** All AI agents (Claude, Gemini, etc.) must follow these rules when working in this repository.
|
||||
|
||||
### File Creation Rules
|
||||
|
||||
1. **No Root Rule** - Never create `.md` files in the repository root unless explicitly instructed by the user.
|
||||
|
||||
2. **Modify, Don't Fork** - Edit existing files; never create `FILE_v2.md`, `FILE_REFERENCE.md`, or `FILE_updated.md` duplicates.
|
||||
|
||||
3. **Scratchpad Protocol** - All analysis, investigation logs, and intermediate work go in `docs/scratch/` with date prefix: `YYYY-MM-DD-<context>.md`
|
||||
|
||||
4. **Consolidation First** - Before creating new docs, search for existing related docs and update them instead.
|
||||
|
||||
### Protected Sections
|
||||
|
||||
Some sections within files are critical and must not be modified without explicit user approval.
|
||||
|
||||
**Inline markers** (source of truth):
|
||||
```markdown
|
||||
<!-- PROTECTED: Do not modify without approval -->
|
||||
## Critical Section Title
|
||||
Content that should not be changed by agents...
|
||||
<!-- END PROTECTED -->
|
||||
```
|
||||
|
||||
For code files:
|
||||
```python
|
||||
# PROTECTED: Do not modify without approval
|
||||
class RPCMethod(Enum):
|
||||
...
|
||||
# END PROTECTED
|
||||
```
|
||||
|
||||
**Rule:** Never modify content between `PROTECTED` and `END PROTECTED` markers unless explicitly instructed by the user.
|
||||
|
||||
### Design Decision Lifecycle
|
||||
|
||||
Design decisions should be captured where they're most useful, not in separate documents that become stale.
|
||||
|
||||
| When | Where | What to Include |
|
||||
|------|-------|-----------------|
|
||||
| **Feature work** | PR description | Design rationale, edge cases, alternatives considered |
|
||||
| **Specific decisions** | Commit message | Why this approach was chosen |
|
||||
| **Large discussions** | GitHub Issue | Link from PR, spans multiple changes |
|
||||
| **Investigation/debugging** | `docs/scratch/` | Temporary work, delete when done |
|
||||
|
||||
**Why not design docs?** Separate design documents accumulate and become stale. PR descriptions stay attached to the code changes, are searchable in GitHub, and don't clutter the repository.
|
||||
|
||||
**Scratch files** (`docs/scratch/`) - Temporary investigation logs and intermediate work. Format: `YYYY-MM-DD-<context>.md`. Periodically cleaned up.
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
| Type | Format | Example |
|
||||
|------|--------|---------|
|
||||
| Root GitHub files | `UPPERCASE.md` | `README.md`, `CONTRIBUTING.md` |
|
||||
| Agent files | `UPPERCASE.md` | `CLAUDE.md`, `AGENTS.md` |
|
||||
| Subfolder README | `README.md` | `docs/adr/README.md` |
|
||||
| All other docs/ files | `lowercase-kebab.md` | `cli-reference.md`, `contributing.md` |
|
||||
| Scratch files | `YYYY-MM-DD-context.md` | `2026-01-06-debug-auth.md` |
|
||||
|
||||
### Status Headers
|
||||
|
||||
All documentation files should include status metadata:
|
||||
|
||||
```markdown
|
||||
**Status:** Active | Deprecated
|
||||
**Last Updated:** YYYY-MM-DD
|
||||
```
|
||||
|
||||
Agents should ignore files marked `Deprecated`.
|
||||
|
||||
### Information Management
|
||||
|
||||
1. **Link, Don't Copy** - Reference README.md sections instead of repeating commands. Prevents drift between docs.
|
||||
|
||||
2. **Scoped Instructions** - Subfolders like `examples/` may have their own README.md with folder-specific rules.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Structure
|
||||
|
||||
```
|
||||
docs/
|
||||
├── adr/ # Architectural Decision Records (ADRs)
|
||||
├── architecture.md # Layered architecture and repository map
|
||||
├── auth-cookie-lifecycle.md # Cookie expiration mitigation strategies and keepalive loops
|
||||
├── cli-exit-codes.md # CLI exit-code convention (binding contract for scripts/CI)
|
||||
├── cli-reference.md # CLI command reference
|
||||
├── configuration.md # Storage, profiles, and settings
|
||||
├── deprecations.md # Staged API deprecations tracker
|
||||
├── development.md # Architecture, testing, and VCR cassette practices
|
||||
├── installation.md # Canonical install guide (personas, extras, platform notes)
|
||||
├── mcp-guide.md # MCP server setup, tools, and troubleshooting
|
||||
├── python-api.md # Python API reference
|
||||
├── refactor-history.md # Historical record of the Tier 12/13 refactor + downstream migration tables
|
||||
├── releasing.md # Release checklist
|
||||
├── rpc-development.md # RPC capture and debugging
|
||||
├── rpc-reference.md # RPC payload structures and Content Type Codes
|
||||
├── stability.md # API versioning and stability policy
|
||||
└── troubleshooting.md # Common issues and solutions
|
||||
```
|
||||
|
||||
Runnable example scripts live at the repository root under `examples/`.
|
||||
|
||||
> When adding or modifying a CLI command, follow the [CLI Exit-Code Convention](docs/cli-exit-codes.md) — the policy table and the two intentional exceptions (`source stale`, `source wait`) are binding.
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 Teng Lin
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,319 @@
|
||||
# notebooklm-py
|
||||
<p align="left">
|
||||
<img src="https://raw.githubusercontent.com/teng-lin/notebooklm-py/main/notebooklm-py.png" alt="notebooklm-py logo" width="128">
|
||||
</p>
|
||||
|
||||
**A Comprehensive NotebookLM Skill & Unofficial Python API.** Full programmatic access to NotebookLM's features—including capabilities the web UI doesn't expose—via Python, CLI, and AI agents like Claude Code, Codex, and OpenClaw.
|
||||
|
||||
[](https://pypi.org/project/notebooklm-py/)
|
||||
[](https://pypi.org/project/notebooklm-py/)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://github.com/teng-lin/notebooklm-py/actions/workflows/test.yml)
|
||||
<p>
|
||||
<a href="https://trendshift.io/repositories/19116" target="_blank"><img src="https://trendshift.io/api/badge/repositories/19116" alt="teng-lin%2Fnotebooklm-py | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>
|
||||
</p>
|
||||
|
||||
**Source & Development**: <https://github.com/teng-lin/notebooklm-py>
|
||||
|
||||
> **⚠️ Unofficial Library - Use at Your Own Risk**
|
||||
>
|
||||
> This library uses **undocumented Google APIs** that can change without notice.
|
||||
>
|
||||
> - **Not affiliated with Google** - This is a community project
|
||||
> - **APIs may break** - Google can change internal endpoints anytime
|
||||
> - **Rate limits apply** - Heavy usage may be throttled
|
||||
>
|
||||
> Best for prototypes, research, and personal projects. See [Troubleshooting](docs/troubleshooting.md) for debugging tips.
|
||||
|
||||
## What You Can Build
|
||||
|
||||
🤖 **AI Agent Tools** - Integrate NotebookLM into Claude Code, Codex, and other LLM agents. Ships with a root [NotebookLM skill](SKILL.md) for GitHub and `npx skills add` discovery, local `notebooklm skill install` support for Claude Code and `.agents` skill directories, and repo-level Codex guidance in [`AGENTS.md`](AGENTS.md).
|
||||
|
||||
📚 **Research Automation** - Bulk-import sources (URLs, PDFs, YouTube, Google Drive), run web/Drive research queries with auto-import, and extract insights programmatically. Build repeatable research pipelines.
|
||||
|
||||
🎙️ **Content Generation** - Generate Audio Overviews (podcasts), videos, slide decks, quizzes, flashcards, infographics, data tables, mind maps, and study guides. Full control over formats, styles, and output.
|
||||
|
||||
📥 **Downloads & Export** - Download all generated artifacts locally (MP3, MP4, PDF, PNG, CSV, JSON, Markdown). Export to Google Docs/Sheets. **Features the web UI doesn't offer**: batch downloads, quiz/flashcard export in multiple formats, mind map JSON extraction.
|
||||
|
||||
## Use Cases & Recipes
|
||||
|
||||
NotebookLM is a **grounded** engine: Gemini does the heavy reading and answers from *your* sources with citations. The winning pattern is to let it do the expensive analysis while your agent (Claude Code, Codex, …) orchestrates and handles the final mile — using NotebookLM as a **zero-token synthesis + memory layer an agent drives in a loop**, and pulling structured artifacts **out** in bulk and in richer, scriptable formats. Recipes people build on top of this library, grouped by what they use NotebookLM *as*:
|
||||
|
||||
**Spend fewer tokens** — let NotebookLM do the expensive thinking:
|
||||
|
||||
- **🪙 Zero-token research offload** — Throw 30 documents into a notebook, let Gemini do the heavy analysis, and have your agent spend tokens only on the final polish. The agent just orchestrates (`create` → `source add` → `ask`); the reasoning happens server-side. *In the wild: [a four-workflow guide to stop Claude Code burning tokens on NotebookLM](https://x.com/hooeem/status/2042293751805329445).*
|
||||
- **🧠 Knowledge distillation → a permanent skill** — Run [Deep Research](docs/cli-reference.md#source-add-research) (`source add-research "your topic" --mode deep`) or load a doc corpus, let NotebookLM's Gemini condense it, and bake the result into a `SKILL.md` your agent loads at startup — **build once, reuse with zero runtime tokens or network calls**, git-versioned and immune to UI drift. A packaged domain expert without hand-curating sources. (Dumping raw docs into a skill flattens the hierarchy; NotebookLM condensing first is what makes it work.)
|
||||
- **✅ Self-validating skills** — Have NotebookLM generate the *eval set* — a quiz straight from your sources — to grade an agent skill against ground truth instead of test questions you'd bias yourself. Build the skill, run it against the NotebookLM-authored evals, iterate to a pass. *In the wild: [a skill that scored 4/10 on the first pass and 10/10 after one iteration, graded by a NotebookLM-generated quiz](https://x.com/nurijanian/status/2037136490157986277).*
|
||||
|
||||
**Give your agent memory** — persistent, grounded recall:
|
||||
|
||||
- **💾 Persistent cross-session memory** — Keep a "Master Brain" notebook; a wrap-up step appends each session's decisions and fixes as notes (`note create` / `ask --save-as-note`), and a line in your `CLAUDE.md` queries it (`ask`) at the start of the next session. Storage and recall live on Google's infrastructure.
|
||||
- **🧩 Grounded memory for coding agents** — Expose a notebook of your internal docs/RFCs/architecture over the [MCP server](docs/mcp-guide.md) (or plain `ask`) so an agent answers from *your* code with citations rather than plausible-sounding guesses — a zero-infra alternative to standing up your own vector DB and embedding pipeline. *In the wild: [turning a notebook into the source-grounded "project brain" a coding agent consults before it writes code](https://medium.com/@pradeep00271/every-software-project-needs-a-project-brain-5cbc33917160).*
|
||||
- **🪞 Query your own notes / journal** — Load years of daily notes, meeting logs, or a journal and `ask` for **cited** answers *across your own history* — surfacing long-term patterns a keyword search can't (e.g. a weekly summary synthesized from 282 daily notes, every claim linked back to the entry it came from). *In the wild: [chatting with a year of daily notes as a cited knowledge base](https://artemxtech.substack.com/p/notebooklm-has-a-knowledge-graph).*
|
||||
|
||||
**Turn your sources into answers & artifacts** — cited responses, generated media, and exports:
|
||||
|
||||
- **📞 Grounded knowledge base / troubleshooting oracle (RAG)** — Load product docs, FAQs, RFCs, and past tickets, then `ask --json` for **source-grounded, cited** answers for support, on-call, or internal Q&A. Or have an agent point it at an entire fast-moving tool's docs — more than the agent can hold in context — as a **troubleshooting oracle** it queries the moment it hits an error. *In the wild: [OpenClaw drove the library to scrape all 524 pages of `docs.openclaw.ai`, dedupe the duplicate translations, and audit it down to 269 clean sources (missing/extra/duplicate = 0)](https://x.com/onenewbite/status/2024819940327379286).*
|
||||
- **🔁 Multi-format content repurposing** — One source set, every format: `generate audio` (podcast), `generate video`, `generate slide-deck`, plus a `generate report` blog draft, `generate quiz`, and `generate flashcards` — fan a single notebook out across channels.
|
||||
- **📤 Bulk, scriptable exports** — Pull mind maps as JSON, flashcards/quizzes as JSON/Markdown/HTML, data tables as CSV, and reports as Markdown — **in bulk, to local files, straight into Anki, your mind-mapping tool, or a repo** (`download <type>` / `download <type> --all`). The programmatic "get data *out*" half of the library, not just "put sources in."
|
||||
- **🕸️ Obsidian / knowledge-graph sync** — Run the CLI from your vault root so downloaded artifacts (reports, mind-map JSON, transcripts) land as files in your knowledge graph; community skills built on this library even resolve NotebookLM's citation markers into Obsidian `[[wikilinks]]`. Pair with a podcast overview for an audio digest of your notes. *In the wild: ["Claude Code + NotebookLM + Obsidian = GOD MODE"](https://www.youtube.com/watch?v=kU3qYQ7ACMA).*
|
||||
|
||||
**Run it unattended, at scale, or on the go** — scheduled, headless, and remote:
|
||||
|
||||
- **🚨 Incident runbook generator** — On an alert, spin up a notebook of the relevant docs, ask targeted diagnostic questions, and generate a briefing-doc report (`generate report --format briefing-doc --wait`, then `download report`) as an automated runbook.
|
||||
- **📚 Curriculum / study-set builder** — Scrape a syllabus or developer roadmap, create one notebook per topic (with deliberate pacing to dodge rate limits), and bulk-generate podcasts, quizzes, and flashcards for each.
|
||||
- **📰 Scheduled audio briefings** — Pair `auth refresh --quiet` (cron/launchd/systemd) with `generate audio` to publish a fresh personalized briefing to a podcast feed on a schedule.
|
||||
- **📱 NotebookLM from your phone, agent-driven** — Self-host the [remote MCP connector](docs/mcp-guide.md#remote-deployment-docker--a-tunnel) behind a Cloudflare/Tailscale tunnel and add it as a custom connector **on the web** (claude.ai Connectors, or ChatGPT with Developer Mode). Then drive the full toolset — deep research, source ingestion, studio generation, cited Q&A — from the **claude.ai mobile app** on the go (ChatGPT's MCP connectors are web-only), chained with your other MCP tools, instead of app-hopping.
|
||||
|
||||
These combine ordinary library primitives — see the [CLI Reference](docs/cli-reference.md) and [Python API](docs/python-api.md). The agent-side glue (skills, scheduling, vault layout) lives in your own setup, not this package. Per-notebook source counts depend on your Google account tier — split across notebooks if you hit a cap.
|
||||
|
||||
**New here?** Start with a walkthrough: [Claude Code + NotebookLM = CHEAT CODE (video)](https://www.youtube.com/watch?v=usTeU4Uh0iM) · [5 demos + 50 use cases, with prompts](https://aiblewmymind.substack.com/p/notebooklm-claude-code-use-cases).
|
||||
|
||||
## Ways to Use
|
||||
|
||||
| Method | Best For |
|
||||
|--------|----------|
|
||||
| **Python API** | Application integration, async workflows, custom pipelines |
|
||||
| **CLI** | Shell scripts, quick tasks, CI/CD automation |
|
||||
| **MCP Server** | Claude Desktop/Code, Codex, etc. — locally via stdio, or as a self-hosted remote connector (behind a Cloudflare/Tailscale tunnel) reachable from claude.ai and ChatGPT, mobile included. |
|
||||
| **REST Server** | Local automation over guarded HTTP routes without spawning a CLI process per call |
|
||||
| **Agent Integration** | Claude Code, Codex, LLM agents, natural language automation |
|
||||
|
||||
## Features
|
||||
|
||||
### Complete NotebookLM Coverage
|
||||
|
||||
| Category | Capabilities |
|
||||
|----------|--------------|
|
||||
| **Notebooks** | Create, list, rename, delete |
|
||||
| **Sources** | URLs, YouTube, files (PDF, text, Markdown, Word, EPUB, audio, video, images), Google Drive, pasted text; refresh, get guide/fulltext |
|
||||
| **Chat** | Questions, conversation history, custom personas, suggested starter prompts |
|
||||
| **Notes** | Create, list, rename, delete, save chat answers, save conversation history |
|
||||
| **Source Labels** | AI-generated or manual topic labels; add/remove source membership; filter sources by label |
|
||||
| **Research** | Web and Drive research agents (fast/deep modes) with auto-import |
|
||||
| **Sharing** | Public/private links, user permissions (viewer/editor), view level control |
|
||||
|
||||
### Content Generation (All Artifact Types)
|
||||
|
||||
| Type | Options | Download Format |
|
||||
|------|---------|-----------------|
|
||||
| **Audio Overview** | 4 formats (deep-dive, brief, critique, debate), 3 lengths, 50+ languages | MP3 |
|
||||
| **Video Overview** | 4 formats (explainer, brief, cinematic, short), 8 visual styles (+ auto/custom), plus a dedicated `cinematic-video` CLI alias | MP4 |
|
||||
| **Slide Deck** | Detailed or presenter format, adjustable length; individual slide revision | PDF, PPTX |
|
||||
| **Infographic** | 3 orientations, 3 detail levels | PNG |
|
||||
| **Quiz** | Configurable quantity and difficulty | JSON, Markdown, HTML |
|
||||
| **Flashcards** | Configurable quantity and difficulty | JSON, Markdown, HTML |
|
||||
| **Report** | Briefing doc, study guide, blog post, or custom prompt | Markdown |
|
||||
| **Data Table** | Custom structure via natural language | CSV |
|
||||
| **Mind Map** | Hierarchical node tree — **two kinds**: note-backed JSON or the newer interactive studio map (`--kind` / `MindMapKind`) | JSON |
|
||||
|
||||
### Beyond the Web UI
|
||||
|
||||
Programmatic, batch, and local-file capabilities the API/CLI make easy — several in richer formats, or at a scale, than clicking through the web app:
|
||||
|
||||
- **Batch downloads** - Download all artifacts of a type at once
|
||||
- **Quiz/Flashcard export** - Get structured JSON, Markdown, or HTML files
|
||||
- **Mind map data extraction** - Export hierarchical JSON for visualization tools
|
||||
- **Data table CSV export** - Download structured tables as spreadsheets
|
||||
- **Slide deck as PPTX or PDF** - Download editable PowerPoint or PDF files
|
||||
- **Slide revision** - Modify individual slides with natural-language prompts
|
||||
- **Report template customization** - Append extra instructions to built-in format templates
|
||||
- **Save chat history to notes** - Persist a whole Q&A conversation (not just a single answer) as a notebook note
|
||||
- **Source fulltext access** - Retrieve the indexed text content of any source
|
||||
- **Programmatic sharing** - Manage permissions without the UI
|
||||
|
||||
## Installation
|
||||
|
||||
The full install guide — six personas (agent, end-user, library, headless, contributor, power-user), optional extras matrix, platform notes — lives in **[docs/installation.md](docs/installation.md)**.
|
||||
|
||||
**Quickest start** (CLI users and AI agents) — install the CLI with `uv tool` (recommended) or `pipx`:
|
||||
|
||||
```bash
|
||||
uv tool install "notebooklm-py[browser]" # or: pipx install "notebooklm-py[browser]"
|
||||
notebooklm login # first run auto-downloads Chromium (~170 MB), then Google sign-in
|
||||
notebooklm auth check --test --json # verify: expect "status": "ok"
|
||||
```
|
||||
|
||||
**Why `uv tool` / `pipx`?** They install the CLI into its own isolated environment and put `notebooklm` on your `PATH` — no dependency clashes with other tools, a one-line upgrade (`uv tool upgrade notebooklm-py`) or uninstall, and, crucially, they work on modern macOS (Homebrew Python) and Debian/Ubuntu where a system-wide `pip install` is blocked with `error: externally-managed-environment` ([PEP 668](https://peps.python.org/pep-0668/)). No `uv` yet? `curl -LsSf https://astral.sh/uv/install.sh | sh` (or `brew install uv` / `winget install astral-sh.uv`).
|
||||
|
||||
**Prefer plain `pip`?** It works the same **inside a virtualenv** (and directly on Windows, where Python isn't externally-managed):
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
|
||||
pip install "notebooklm-py[browser]"
|
||||
```
|
||||
|
||||
**As a library** (embedded in your app — no Playwright, no Chromium):
|
||||
|
||||
```bash
|
||||
uv add notebooklm-py # or, inside a virtualenv: pip install notebooklm-py
|
||||
```
|
||||
|
||||
If `playwright install chromium` fails on Linux with `TypeError: onExit is not a function`, see the [Linux workaround](docs/troubleshooting.md#linux). **Contributors:** see [CONTRIBUTING.md](CONTRIBUTING.md).
|
||||
|
||||
|
||||
### Authentication & Access
|
||||
|
||||
Flexible auth for local dev, headless servers, and multi-tenant setups:
|
||||
|
||||
- **Three ways to get cookies** - Interactive Playwright login (default), import from an already-signed-in browser (`login --browser-cookies chrome`, no Playwright), or a durable **master token**.
|
||||
- **Master-token auth** - Mints fresh web cookies **on demand** with no per-session browser (`login --master-token --account you@example.com`), so it self-heals expired sessions unattended — the auth model for servers, CI, and the remote MCP connector (claude.ai / ChatGPT).
|
||||
- **Multi-account profiles** - Switch between Google accounts without re-authenticating.
|
||||
|
||||
### Agent Setup
|
||||
|
||||
**Option 1 — CLI install**:
|
||||
|
||||
```bash
|
||||
notebooklm skill install
|
||||
```
|
||||
|
||||
Installs the skill into `~/.claude/skills/notebooklm` and `~/.agents/skills/notebooklm`.
|
||||
|
||||
**Option 2 — `npx` install** (via the open skills ecosystem):
|
||||
|
||||
```bash
|
||||
npx skills add teng-lin/notebooklm-py
|
||||
```
|
||||
|
||||
Fetches the canonical [SKILL.md](SKILL.md) directly from GitHub.
|
||||
|
||||
## Quick Start
|
||||
|
||||
<p align="center">
|
||||
<a href="https://asciinema.org/a/767284" target="_blank"><img src="https://asciinema.org/a/767284.svg" width="600" /></a>
|
||||
<br>
|
||||
<em>16-minute session compressed to 30 seconds</em>
|
||||
</p>
|
||||
|
||||
### CLI
|
||||
|
||||
```bash
|
||||
# 1. Authenticate (opens browser)
|
||||
notebooklm login
|
||||
# Or use Microsoft Edge (for orgs that require Edge for SSO)
|
||||
# notebooklm login --browser msedge
|
||||
# Or reuse cookies from an already-logged-in browser session
|
||||
# notebooklm login --browser-cookies chrome
|
||||
# notebooklm login --browser-cookies 'chrome::Profile 1' # one Chromium profile
|
||||
# (combine with --profile to populate a specific profile;
|
||||
# use --account / --all-accounts after auth inspect when several
|
||||
# Google accounts are signed in)
|
||||
|
||||
# 2. Create a notebook and add sources
|
||||
notebooklm create "My Research"
|
||||
notebooklm use <notebook_id>
|
||||
notebooklm source add "https://en.wikipedia.org/wiki/Artificial_intelligence"
|
||||
notebooklm source add "./paper.pdf"
|
||||
|
||||
# 3. Chat with your sources
|
||||
notebooklm ask "What are the key themes?"
|
||||
notebooklm ask --prompt-file ./long_question.txt # Read question from file
|
||||
|
||||
# 4. Generate content (use --prompt-file for long prompts)
|
||||
notebooklm generate audio "make it engaging" --wait
|
||||
notebooklm generate video --style whiteboard --wait
|
||||
notebooklm generate cinematic-video "documentary-style summary" --wait
|
||||
notebooklm generate quiz --difficulty hard
|
||||
notebooklm generate flashcards --quantity more
|
||||
notebooklm generate slide-deck
|
||||
notebooklm generate infographic --orientation portrait
|
||||
notebooklm generate mind-map # interactive studio map (default); --kind note-backed for the JSON tree
|
||||
notebooklm generate data-table "compare key concepts"
|
||||
|
||||
# 5. Download artifacts
|
||||
notebooklm download audio ./podcast.mp3
|
||||
notebooklm download video ./overview.mp4
|
||||
notebooklm download cinematic-video ./documentary.mp4
|
||||
notebooklm download quiz --format markdown ./quiz.md
|
||||
notebooklm download flashcards --format json ./cards.json
|
||||
notebooklm download slide-deck ./slides.pdf
|
||||
notebooklm download infographic ./infographic.png
|
||||
notebooklm download mind-map ./mindmap.json
|
||||
notebooklm download data-table ./data.csv
|
||||
```
|
||||
|
||||
Other useful CLI commands:
|
||||
|
||||
```bash
|
||||
notebooklm auth check --test # Diagnose auth/cookie issues
|
||||
notebooklm auth refresh --quiet # One-shot cookie keepalive (for cron / launchd / systemd)
|
||||
notebooklm auth refresh --browser-cookies chrome # Re-extract and repair account routing
|
||||
notebooklm auth inspect --browser 'chrome::Profile 1' # Preview one Chromium profile
|
||||
notebooklm agent show codex # Print bundled Codex instructions
|
||||
notebooklm agent show claude # Print bundled Claude Code skill template
|
||||
notebooklm language list # List supported output languages
|
||||
notebooklm metadata --json # Export notebook metadata and sources
|
||||
notebooklm share status # Inspect sharing state
|
||||
notebooklm source add-research "AI" --import-all # web research + import found sources
|
||||
notebooklm skill status # Check local agent skill installation
|
||||
notebooklm profile list # List all Google account profiles
|
||||
notebooklm profile switch work # Switch active account profile
|
||||
```
|
||||
|
||||
Use `--prompt-file PATH` with `ask`, prompt-based `generate` commands, and `source add-research` when the text is too long for the shell command line. This reads prompt/query text from a file and is separate from `source add ./file.pdf`, which still uploads that file as a NotebookLM source.
|
||||
|
||||
### Python API
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from notebooklm import NotebookLMClient, MindMapKind
|
||||
|
||||
async def main():
|
||||
async with NotebookLMClient.from_storage() as client:
|
||||
# Create notebook and add sources
|
||||
nb = await client.notebooks.create("Research")
|
||||
await client.sources.add_url(nb.id, "https://example.com", wait=True)
|
||||
|
||||
# Chat with your sources
|
||||
result = await client.chat.ask(nb.id, "Summarize this")
|
||||
print(result.answer)
|
||||
|
||||
# Generate content (podcast, video, quiz, etc.)
|
||||
status = await client.artifacts.generate_audio(nb.id, instructions="make it fun")
|
||||
await client.artifacts.wait_for_completion(nb.id, status.task_id)
|
||||
await client.artifacts.download_audio(nb.id, "podcast.mp3")
|
||||
|
||||
# Generate quiz and download as JSON
|
||||
status = await client.artifacts.generate_quiz(nb.id)
|
||||
await client.artifacts.wait_for_completion(nb.id, status.task_id)
|
||||
await client.artifacts.download_quiz(nb.id, "quiz.json", output_format="json")
|
||||
|
||||
# Generate a mind map via the unified client.mind_maps API (issue #1256) —
|
||||
# two kinds: the newer MindMapKind.INTERACTIVE studio map (shown; polled to
|
||||
# completion by default) or MindMapKind.NOTE_BACKED JSON. Both export via:
|
||||
mm = await client.mind_maps.generate(nb.id, kind=MindMapKind.INTERACTIVE)
|
||||
await client.artifacts.download_mind_map(nb.id, "mindmap.json", mm.id)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
- **[CLI Reference](docs/cli-reference.md)** - Complete command documentation
|
||||
- **[Python API](docs/python-api.md)** - Full API reference
|
||||
- **[MCP Guide](docs/mcp-guide.md)** - MCP server setup, transports, and tool reference
|
||||
- **[REST API Server](docs/installation.md#rest-api-server)** - Experimental localhost FastAPI server
|
||||
- **[Configuration](docs/configuration.md)** - Storage and settings
|
||||
- **[Quota & Tier Limits](docs/quota-limits.md)** - Per-tier notebook/source/studio limits and how they map to `AccountLimits.tier`
|
||||
- **[Release Guide](docs/releasing.md)** - Release checklist and packaging verification
|
||||
- **[Troubleshooting](docs/troubleshooting.md)** - Common issues and solutions
|
||||
- **[API Stability](docs/stability.md)** - Versioning policy and stability guarantees
|
||||
- **[Upgrading to v0.8.0](docs/upgrading-to-0.8.0.md)** - Breaking-change migration guide for the v0.8.0 error-and-return contract
|
||||
|
||||
### For Contributors
|
||||
|
||||
- **[Architecture](docs/architecture.md)** - Architectural overview and design principles
|
||||
- **[Development Guide](docs/development.md)** - Architecture, testing, and releasing
|
||||
- **[RPC Development](docs/rpc-development.md)** - Protocol capture and debugging
|
||||
- **[RPC Reference](docs/rpc-reference.md)** - Payload structures
|
||||
- **[Changelog](CHANGELOG.md)** - Version history and release notes
|
||||
- **[Security](SECURITY.md)** - Security policy and credential handling
|
||||
|
||||
## Star History
|
||||
|
||||
[](https://www.star-history.com/?repos=teng-lin%2Fnotebooklm-py&type=timeline&legend=top-left)
|
||||
|
||||
## License
|
||||
|
||||
MIT License. See [LICENSE](LICENSE) for details.
|
||||
@@ -0,0 +1,7 @@
|
||||
# WeHub 来源说明
|
||||
|
||||
- 原始项目:`teng-lin/notebooklm-py`
|
||||
- 原始仓库:https://github.com/teng-lin/notebooklm-py
|
||||
- 导入方式:上游默认分支的最新快照
|
||||
- 原作者、版权和许可证信息以原始仓库及本仓库 LICENSE 为准
|
||||
- 本文件仅用于记录来源,不代表 WeHub 是原项目作者
|
||||
+129
@@ -0,0 +1,129 @@
|
||||
# Security Policy
|
||||
|
||||
## Supported Versions
|
||||
|
||||
Only the latest minor release receives security fixes. Earlier `0.x` releases predate the current API surface and are no longer maintained — please upgrade to the latest version.
|
||||
|
||||
| Version | Supported |
|
||||
| ------- | ------------------ |
|
||||
| 0.8.x | :white_check_mark: |
|
||||
| < 0.8 | :x: |
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
If you discover a security vulnerability, please report it by:
|
||||
|
||||
1. **DO NOT** open a public GitHub issue
|
||||
2. Email the maintainers directly (see `pyproject.toml` for contact info)
|
||||
3. Include a detailed description of the vulnerability
|
||||
4. Allow reasonable time for a fix before public disclosure
|
||||
|
||||
## Credential Security
|
||||
|
||||
This library stores authentication credentials locally. Please understand these security considerations:
|
||||
|
||||
### Storage Locations
|
||||
|
||||
By default, files are stored per-profile under `~/.notebooklm/profiles/<profile>/` (configurable via the `NOTEBOOKLM_HOME` and `NOTEBOOKLM_PROFILE` environment variables). Legacy layouts store files directly in the root of `~/.notebooklm/` (representing the `default` profile). Permission modes below are POSIX modes; Windows uses the inherited filesystem ACLs and intentionally skips `chmod`.
|
||||
|
||||
| File Path | Contents | Permissions |
|
||||
|-----------|----------|-------------|
|
||||
| `profiles/<profile>/storage_state.json` | Google session cookies | `0o600` (owner-only) |
|
||||
| `profiles/<profile>/browser_profile/` | Playwright Chromium profile | `0o700` (owner-only) |
|
||||
| `profiles/<profile>/context.json` | Active profile context / metadata | `0o600` (owner-only) |
|
||||
| `config.json` | Global CLI config (e.g. language/active profile) | Default |
|
||||
| `storage_state.json` *(legacy)* | Fallback root storage state (for `default` profile) | `0o600` (owner-only) |
|
||||
| `browser_profile/` *(legacy)* | Fallback root Playwright profile | `0o700` (owner-only) |
|
||||
| `context.json` *(legacy)* | Fallback root active notebook context | `0o600` (owner-only, POSIX) |
|
||||
|
||||
### Security Best Practices
|
||||
|
||||
1. **Protect your credentials**
|
||||
- The `storage_state.json` file contains your Google session cookies
|
||||
- Anyone with access to this file can impersonate your Google account to NotebookLM
|
||||
- Never share, commit, or expose this file
|
||||
|
||||
2. **Add to .gitignore**
|
||||
```gitignore
|
||||
.notebooklm/
|
||||
```
|
||||
|
||||
3. **Credential rotation**
|
||||
- Re-run `notebooklm login` periodically to refresh credentials
|
||||
- Sessions typically last days to weeks before expiring
|
||||
|
||||
4. **If credentials are compromised**
|
||||
- Immediately revoke access at [Google Security Settings](https://myaccount.google.com/permissions)
|
||||
- Delete the `~/.notebooklm/` directory
|
||||
- Re-authenticate with `notebooklm login`
|
||||
|
||||
5. **CI/CD usage**
|
||||
- Do not commit credentials to repositories
|
||||
- Use `NOTEBOOKLM_AUTH_JSON` environment variable for secure, file-free authentication
|
||||
- Store the JSON value in GitHub Secrets or similar secure secret management
|
||||
- The env var approach keeps credentials in memory only, never written to disk
|
||||
|
||||
### What This Library Does NOT Do
|
||||
|
||||
- Does not transmit credentials to any third party
|
||||
- Does not store passwords (uses browser-based OAuth)
|
||||
- Does not access data outside of NotebookLM except for user-selected local files
|
||||
and opt-in browser-cookie extraction during login/refresh
|
||||
- Does not modify Google account settings
|
||||
|
||||
## Dependency Security
|
||||
|
||||
This library keeps the base dependency set small and puts optional surfaces behind
|
||||
extras:
|
||||
|
||||
| Dependency | Scope | Purpose |
|
||||
|------------|-------|---------|
|
||||
| `httpx` | base | HTTP client |
|
||||
| `click` | base | CLI framework |
|
||||
| `rich` | base | Terminal output |
|
||||
| `filelock` | base | Cross-process file locking for profile/context writes |
|
||||
| `markdownify` | `markdown` extra | HTML-to-Markdown conversion |
|
||||
| `playwright` | `browser` extra | Interactive/headless browser login |
|
||||
| `rookiepy` | `cookies` extra | Opt-in browser-cookie import |
|
||||
| `fastmcp` | `mcp` extra | MCP server adapter |
|
||||
| `fastapi`, `uvicorn[standard]`, `python-multipart` | `server` extra | Optional REST server and file uploads |
|
||||
|
||||
### Auditing Dependencies
|
||||
|
||||
```bash
|
||||
# Mirror CI: audit the locked selected-extra graph
|
||||
uv sync --frozen --extra browser --extra dev --extra markdown
|
||||
uv run python -m pip install "pip-audit>=2.7.0,<3"
|
||||
uv export --frozen --extra browser --extra dev --extra markdown --format requirements-txt --no-emit-project \
|
||||
| uv run pip-audit --strict --require-hashes --disable-pip -r /dev/stdin
|
||||
|
||||
# Release/security sweep: include maintained MCP + REST server extras too
|
||||
uv export --frozen --extra browser --extra dev --extra markdown --extra mcp --extra server \
|
||||
--format requirements-txt --no-emit-project \
|
||||
| uv run pip-audit --strict --require-hashes --disable-pip -r /dev/stdin
|
||||
```
|
||||
|
||||
The `cookies` extra remains an explicit opt-in because `rookiepy` has had
|
||||
interpreter compatibility issues; audit that graph separately when changing the
|
||||
browser-cookie import surface.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
### Undocumented API
|
||||
|
||||
This library uses Google's internal APIs, which means:
|
||||
|
||||
- **No official security guarantees** from Google
|
||||
- **API changes without notice** may break functionality
|
||||
- **Rate limiting** may be applied by Google
|
||||
- **Account restrictions** are possible for unusual usage patterns
|
||||
|
||||
### Session Security
|
||||
|
||||
- Sessions are cookie-based (standard web authentication)
|
||||
- CSRF tokens are required and automatically handled
|
||||
- No long-lived API keys or OAuth tokens
|
||||
|
||||
## Questions?
|
||||
|
||||
For security questions that are not vulnerabilities, open a [GitHub Discussion](https://github.com/teng-lin/notebooklm-py/discussions).
|
||||
@@ -0,0 +1,708 @@
|
||||
---
|
||||
name: notebooklm
|
||||
description: Complete API for Google NotebookLM - full programmatic access including features not in the web UI. Create notebooks, add sources, generate all artifact types, download in multiple formats. Activates on explicit /notebooklm or intent like "create a podcast about X"
|
||||
---
|
||||
|
||||
# NotebookLM Automation
|
||||
|
||||
Complete programmatic access to Google NotebookLM—including capabilities not exposed in the web UI. Create notebooks, add sources (URLs, YouTube, PDFs, audio, video, images), chat with content, generate all artifact types, and download results in multiple formats.
|
||||
|
||||
## Installation
|
||||
|
||||
**From PyPI (Recommended for AI agents — Python-version-aware):**
|
||||
```bash
|
||||
pip install "notebooklm-py[browser]" # mandatory; errors must propagate
|
||||
|
||||
# [cookies] (rookiepy) is optional and known to FAIL TO BUILD on Python 3.13+.
|
||||
# Skip it deliberately on 3.13+ rather than swallowing the error — that lets
|
||||
# *real* install failures (typos, network, PyPI outages) surface for the agent.
|
||||
if python -c "import sys; sys.exit(0 if sys.version_info < (3, 13) else 1)"; then
|
||||
pip install "notebooklm-py[cookies]" # errors propagate
|
||||
else
|
||||
echo "Skipping [cookies] on Python 3.13+ (rookiepy unavailable). Use 'notebooklm login' interactively."
|
||||
fi
|
||||
```
|
||||
|
||||
> Full install matrix (extras, headless servers, contributor flow): [Installation guide on GitHub](https://github.com/teng-lin/notebooklm-py/blob/main/docs/installation.md).
|
||||
|
||||
**From GitHub (use latest release tag, NOT main branch):**
|
||||
```bash
|
||||
# Get the latest release tag (requires curl + jq)
|
||||
if ! command -v jq >/dev/null; then
|
||||
echo "jq is required to read the latest release tag" >&2
|
||||
exit 1
|
||||
fi
|
||||
LATEST_TAG=$(
|
||||
curl -fsSL https://api.github.com/repos/teng-lin/notebooklm-py/releases/latest |
|
||||
jq -r '.tag_name'
|
||||
)
|
||||
# Includes [browser] so the interactive `notebooklm login` flow works.
|
||||
pip install "notebooklm-py[browser] @ git+https://github.com/teng-lin/notebooklm-py@${LATEST_TAG}"
|
||||
```
|
||||
|
||||
⚠️ **DO NOT install from main branch** (`pip install git+https://github.com/teng-lin/notebooklm-py`). The main branch may contain unreleased/unstable changes. Always use PyPI or a specific release tag, unless you are testing unreleased features.
|
||||
|
||||
**Skill install methods:**
|
||||
|
||||
- `notebooklm skill install` installs this skill into the supported local agent directories managed by the CLI.
|
||||
- `npx skills add teng-lin/notebooklm-py` installs this skill from the GitHub repository into compatible agent skill directories.
|
||||
- If you are already reading this file inside an agent skill directory, the skill is already installed. You only need the Python package and authentication below.
|
||||
|
||||
**CLI-managed install:**
|
||||
```bash
|
||||
notebooklm skill install
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
**IMPORTANT:** Before using any command, you MUST authenticate:
|
||||
|
||||
```bash
|
||||
notebooklm login # Opens browser for Google OAuth
|
||||
notebooklm list # Verify authentication works
|
||||
```
|
||||
|
||||
If commands fail with authentication errors, re-run `notebooklm login`.
|
||||
|
||||
### CI/CD, Multiple Accounts, and Parallel Agents
|
||||
|
||||
For automated environments, multiple accounts, or parallel agent workflows:
|
||||
|
||||
| Variable | Purpose |
|
||||
|----------|---------|
|
||||
| `NOTEBOOKLM_HOME` | Custom config directory (default: `~/.notebooklm`) |
|
||||
| `NOTEBOOKLM_PROFILE` | Active profile name (default: `default`) |
|
||||
| `NOTEBOOKLM_AUTH_JSON` | Inline auth JSON - no file writes needed |
|
||||
|
||||
**CI/CD setup:** Set `NOTEBOOKLM_AUTH_JSON` from a secret containing your `storage_state.json` contents.
|
||||
|
||||
**Multiple accounts:** Use named profiles (`notebooklm profile create work`, then `notebooklm -p work login`). Alternatively, use different `NOTEBOOKLM_HOME` directories per account.
|
||||
|
||||
**Parallel agents:** The CLI stores notebook context per profile (`~/.notebooklm/profiles/<profile>/context.json`, with a legacy fallback to `~/.notebooklm/context.json` for the implicit default profile). Multiple concurrent agents that share a profile and use `notebooklm use` can overwrite each other's context — use one of the isolation strategies below.
|
||||
|
||||
**Solutions for parallel workflows:**
|
||||
1. **Always use explicit notebook ID** (recommended): Pass `-n <notebook_id>` / `--notebook <notebook_id>` on notebook-scoped commands instead of relying on `use`
|
||||
2. **Per-agent isolation via profiles:** `export NOTEBOOKLM_PROFILE=agent-$ID` (each profile gets its own context file)
|
||||
3. **Per-agent isolation via home:** Set unique `NOTEBOOKLM_HOME` per agent: `export NOTEBOOKLM_HOME=/tmp/agent-$ID`
|
||||
4. **Use full UUIDs:** Avoid partial IDs in automation (they can become ambiguous)
|
||||
|
||||
### Sandboxed Agents (Claude Cowork / Headless)
|
||||
|
||||
Sandboxed, no-display agent environments — **Claude Cowork** (Anthropic's desktop agent for non-developers) and similar headless sandboxes — can't run `notebooklm login` (it needs a browser), and they reset between sessions. Everything else works with two adjustments:
|
||||
|
||||
1. **Bootstrap each session.** The sandbox resets, so install at the start of every session. You do **not** need `[browser]`/Playwright here — that extra exists only for the interactive `login` flow, which you run on a host machine, not in the sandbox. Chat, sources, generation, and download all run on the base install:
|
||||
```bash
|
||||
pip install notebooklm-py # no [browser] needed for queries/generation
|
||||
```
|
||||
(This is the one place the mandatory `[browser]` install at the top of this file does not apply — you are reusing auth, not logging in here.)
|
||||
2. **Reuse a host-generated `storage_state.json`.** Log in once on a machine with a display (`notebooklm login`), then bring the resulting `storage_state.json` into a sandbox-accessible folder and point at it either way:
|
||||
```bash
|
||||
# Per-invocation root flag (a persistent, sandbox-accessible path):
|
||||
notebooklm --storage /path/to/storage_state.json list
|
||||
# OR inline via env var (no file needed — e.g. from a Cowork-stored secret):
|
||||
export NOTEBOOKLM_AUTH_JSON="$(cat /path/to/storage_state.json)"
|
||||
notebooklm list
|
||||
```
|
||||
|
||||
> ⚠️ **`storage_state.json` / `NOTEBOOKLM_AUTH_JSON` are bearer credentials** — anyone holding them can act as your Google account. Keep the file `0600`, load it from the sandbox's secret store rather than a committed file, never print or log it, and `unset NOTEBOOKLM_AUTH_JSON` when finished.
|
||||
|
||||
**Verify** as in [Agent Setup Verification](#agent-setup-verification) below — e.g. `notebooklm --storage <path> auth check --test --json` (require `"status": "ok"` AND `"checks.token_fetch": true`).
|
||||
|
||||
**Context does not survive a reset** either: the selected-notebook context (`context.json`) is gone each session, so pass an explicit `-n/--notebook <id>` on notebook-scoped commands instead of relying on `notebooklm use`.
|
||||
|
||||
If Cowork reads `~/.claude/skills/`, `notebooklm skill install` registers this skill there automatically; otherwise build the uploadable archive on the host with `notebooklm skill package` (writes `notebooklm-skill.zip`) and add it via **Claude Settings → Capabilities**. Full recipe (extras matrix, headless auth, CI env-var notes): [installation.md § AI Agent](https://github.com/teng-lin/notebooklm-py/blob/main/docs/installation.md#a-ai-agent-primary-persona).
|
||||
|
||||
## Agent Setup Verification
|
||||
|
||||
Before starting workflows, verify auth is in place. **Use `--test --json` (not bare `--json`)** — bare `--json` only proves the cookie file parses; `--test` makes a network call and proves the cookies still authenticate against Google.
|
||||
|
||||
1. `notebooklm auth check --test --json` → require BOTH `"status": "ok"` AND `"checks.token_fetch": true`. Bare `"status": "ok"` (without `--test`) is a false-positive trap — a stale cookie file passes the parse check.
|
||||
2. `notebooklm list --json` → expect valid JSON (may be empty for new accounts).
|
||||
3. **If auth fails or is missing → run `notebooklm login` first.** This is the primary auth path: opens a browser, the user signs in to Google once, and the resulting `storage_state.json` is reused on every subsequent run. Works on any environment with a display.
|
||||
- For headless contexts where opening a browser is not feasible, use `notebooklm login --browser-cookies <browser>` instead — extracts the user's already-logged-in cookies from Chrome/Firefox/etc. (requires the `[cookies]` extra; rookiepy may not install on Python 3.13+). Use `chrome::<profile-name-or-directory>` to target one Chromium user-profile, or `firefox::<container-name>` / `firefox::none` to target one Firefox container.
|
||||
- To survey signed-in Google accounts before picking one: `notebooklm auth inspect --browser <browser>` (read-only; pass `-v` to see which Chromium user-profile each account came from, or `--json` for tooling). Scoped forms such as `notebooklm auth inspect --browser 'chrome::Profile 1'` inspect only that browser profile.
|
||||
- Re-run step 1 after login to confirm.
|
||||
4. **If auth was working but cookies went stale** (Google rotated SIDTS, or you signed in fresh in the browser) **→ refresh the active profile in place instead of full re-login:**
|
||||
- `notebooklm auth refresh` — server-side SIDTS refresh against the existing `storage_state.json`. Cheap and silent; safe to run on a schedule (cron / launchd / systemd) at 15–20 min cadence to keep an unattended profile warm.
|
||||
- `notebooklm auth refresh --browser-cookies <browser>` — re-extract cookies from a running browser and match them back to the profile's recorded email in `context.json`. Use when the on-disk `storage_state.json` is too stale for the server-side refresh path but you've just signed back into Google in the browser. For Chromium-family browsers with multiple user-profiles (Chrome's `Default`, `Profile 1`, …), refresh fans out across all profiles to find the email — same path as `auth inspect` (issue #571). Use `chrome::<profile-name-or-directory>` when you already know the exact browser profile.
|
||||
- Both forms preserve the same `--profile` (no new profile is created).
|
||||
|
||||
> **Note:** `notebooklm status` reports *context state* (selected notebook); do not use it to verify auth.
|
||||
|
||||
## When This Skill Activates
|
||||
|
||||
**Explicit:** User says "/notebooklm", "use notebooklm", or mentions the tool by name
|
||||
|
||||
**Intent detection:** Recognize requests like:
|
||||
- "Create a podcast about [topic]"
|
||||
- "Summarize these URLs/documents"
|
||||
- "Generate a quiz from my research"
|
||||
- "Turn this into an audio overview"
|
||||
- "Create flashcards for studying"
|
||||
- "Generate a video explainer"
|
||||
- "Make an infographic"
|
||||
- "Create a mind map of the concepts"
|
||||
- "Download the quiz as markdown"
|
||||
- "Add these sources to NotebookLM"
|
||||
|
||||
## Autonomy Rules
|
||||
|
||||
**Run automatically (no confirmation):**
|
||||
- `notebooklm status` - check context
|
||||
- `notebooklm auth check` - diagnose auth issues
|
||||
- `notebooklm auth inspect` - list Google accounts visible to a browser (read-only)
|
||||
- `notebooklm auth refresh` - server-side SIDTS refresh of the active profile (no new profile, no destructive writes)
|
||||
- `notebooklm auth refresh --browser-cookies <browser>` - re-extract cookies from a browser into the active profile (rebuilds `storage_state.json` for the same `--profile`, not a new one)
|
||||
- `notebooklm list` - list notebooks
|
||||
- `notebooklm source list` - list sources
|
||||
- `notebooklm artifact list` - list artifacts
|
||||
- `notebooklm language list` - list supported languages
|
||||
- `notebooklm language get` - get current language
|
||||
- `notebooklm language set` - set language (global setting)
|
||||
- `notebooklm artifact wait` - wait for artifact completion (in subagent context)
|
||||
- `notebooklm source wait` - wait for source processing (in subagent context)
|
||||
- `notebooklm research status` - check research status
|
||||
- `notebooklm research wait` - wait for research (in subagent context)
|
||||
- `notebooklm use <id>` - set context (⚠️ SINGLE-AGENT ONLY - use `-n` flag in parallel workflows)
|
||||
- `notebooklm create` - create notebook
|
||||
- `notebooklm ask "..."` - chat queries (without `--save-as-note`)
|
||||
- `notebooklm suggest-prompts` - AI-suggested prompts for a notebook (read-only, no state change)
|
||||
- `notebooklm history` - display conversation history (read-only)
|
||||
- `notebooklm source add` - add sources
|
||||
- `notebooklm profile list` - list profiles
|
||||
- `notebooklm profile create` - create profile
|
||||
- `notebooklm profile switch` - switch active profile
|
||||
- `notebooklm doctor` - check environment health
|
||||
|
||||
**Ask before running:**
|
||||
- `notebooklm delete`, `source delete`, `source delete-by-title`, `source clean`, `note delete`, `artifact delete`, `label delete`, `share remove`, `auth logout`, `clear`, `profile delete`, or `ask --new` - destructive or state-changing. Once approved, pass `--yes`/`-y` where the command supports it. Most destructive `--json` commands still require explicit `--yes` and otherwise return a structured confirmation error (`CONFIRM_REQUIRED` or `VALIDATION_ERROR`, depending on the command family); current exceptions include `share remove --json` and `ask --new --json`, which skip the prompt for non-interactive callers.
|
||||
- `notebooklm generate *` - long-running, may fail
|
||||
- `notebooklm download *` - writes to filesystem
|
||||
- `notebooklm artifact wait` - long-running (when in main conversation)
|
||||
- `notebooklm source wait` - long-running (when in main conversation)
|
||||
- `notebooklm research wait` - long-running (when in main conversation)
|
||||
- `notebooklm research cancel <run_id>` - state-changing; cancels a running research job (an in-progress job transitions to FAILED). Fire-and-forget: it does not confirm success — re-check with `notebooklm research status`.
|
||||
- `notebooklm ask "..." --save-as-note` - writes a note
|
||||
- `notebooklm history --save` - writes a note
|
||||
|
||||
## Quick Reference
|
||||
|
||||
| Task | Command |
|
||||
|------|---------|
|
||||
| Authenticate | `notebooklm login` |
|
||||
| Authenticate from browser cookies | `notebooklm login --browser-cookies <browser>` |
|
||||
| Authenticate from one Chromium profile | `notebooklm login --browser-cookies 'chrome::Profile 1'` |
|
||||
| Authenticate from one Firefox container | `notebooklm login --browser-cookies 'firefox::Work'` |
|
||||
| Import every signed-in account into its own profile | `notebooklm login --browser-cookies <browser> --all-accounts` |
|
||||
| Inspect signed-in accounts (read-only, by email) | `notebooklm auth inspect --browser <browser>` |
|
||||
| Inspect one browser profile/container | `notebooklm auth inspect --browser 'chrome::Profile 1'` |
|
||||
| Diagnose auth issues | `notebooklm auth check` |
|
||||
| Diagnose auth (full) | `notebooklm auth check --test` |
|
||||
| Refresh active profile in place (server-side) | `notebooklm auth refresh` |
|
||||
| Refresh active profile from a re-signed-in browser | `notebooklm auth refresh --browser-cookies <browser>` |
|
||||
| Refresh from one Chromium profile | `notebooklm auth refresh --browser-cookies 'chrome::Profile 1'` |
|
||||
| One-shot cookie keepalive (for cron) | `notebooklm auth refresh --quiet` |
|
||||
| List notebooks | `notebooklm list` |
|
||||
| Create notebook | `notebooklm create "Title"` |
|
||||
| Set context | `notebooklm use <notebook_id>` |
|
||||
| Show context | `notebooklm status` |
|
||||
| Add URL source | `notebooklm source add "https://..."` |
|
||||
| Add file | `notebooklm source add ./file.pdf` |
|
||||
| Add YouTube | `notebooklm source add "https://youtube.com/..."` |
|
||||
| List sources | `notebooklm source list` |
|
||||
| List sources in a label | `notebooklm source list --label <label_id_or_name>` |
|
||||
| Delete source by ID | `notebooklm source delete <source_id>` |
|
||||
| Delete source by exact title | `notebooklm source delete-by-title "Exact Title"` |
|
||||
| Wait for source processing | `notebooklm source wait <source_id>` |
|
||||
| List labels | `notebooklm label list` |
|
||||
| Expand label to sources | `notebooklm label sources <label_id_or_name>` |
|
||||
| Generate labels | `notebooklm label generate --scope unlabeled` |
|
||||
| Create label | `notebooklm label create "Topic"` |
|
||||
| Add sources to label | `notebooklm label add <label_id_or_name> <source_id>...` |
|
||||
| Remove sources from label | `notebooklm label remove <label_id_or_name> <source_id>...` |
|
||||
| Delete label | `notebooklm label delete <label_id_or_name> --yes` |
|
||||
| Web research (fast) | `notebooklm source add-research "query"` |
|
||||
| Web research (deep) | `notebooklm source add-research "query" --mode deep --no-wait` |
|
||||
| Web research (query from file) | `notebooklm source add-research --prompt-file research_query.txt --mode deep` |
|
||||
| Check research status | `notebooklm research status` |
|
||||
| Wait for research | `notebooklm research wait --import-all` |
|
||||
| Cancel research | `notebooklm research cancel <run_id>` (run_id = the `task_id` from `research status`) |
|
||||
| Suggest questions to ask | `notebooklm suggest-prompts` |
|
||||
| Chat | `notebooklm ask "question"` |
|
||||
| Chat (long prompt from file) | `notebooklm ask --prompt-file question.txt` |
|
||||
| Chat (specific sources) | `notebooklm ask "question" -s src_id1 -s src_id2` |
|
||||
| Chat (with references) | `notebooklm ask "question" --json` |
|
||||
| Chat (save answer as note) | `notebooklm ask "question" --save-as-note` |
|
||||
| Chat (save with title) | `notebooklm ask "question" --save-as-note --note-title "Title"` |
|
||||
| Show conversation history | `notebooklm history` |
|
||||
| Save all history as note | `notebooklm history --save` |
|
||||
| Continue specific conversation | `notebooklm ask "question" -c <conversation_id>` |
|
||||
| Save history with title | `notebooklm history --save --note-title "My Research"` |
|
||||
| Get source fulltext | `notebooklm source fulltext <source_id>` |
|
||||
| Get source guide | `notebooklm source guide <source_id>` |
|
||||
| Generate podcast | `notebooklm generate audio "instructions"` |
|
||||
| Generate (long prompt from file) | `notebooklm generate audio --prompt-file instructions.txt` |
|
||||
| Generate podcast (JSON) | `notebooklm generate audio --json` |
|
||||
| Generate podcast (specific sources) | `notebooklm generate audio -s src_id1 -s src_id2` |
|
||||
| Generate video | `notebooklm generate video "instructions"` |
|
||||
| Generate report | `notebooklm generate report --format briefing-doc` |
|
||||
| Generate report (append instructions) | `notebooklm generate report --format study-guide --append "Target audience: beginners"` |
|
||||
| Generate quiz | `notebooklm generate quiz` |
|
||||
| Revise a slide | `notebooklm generate revise-slide "prompt" --artifact <id> --slide 0` |
|
||||
| Check artifact status | `notebooklm artifact list` |
|
||||
| Wait for completion | `notebooklm artifact wait <artifact_id>` |
|
||||
| Delete artifact | `notebooklm artifact delete <artifact_id> --yes` |
|
||||
| Download audio | `notebooklm download audio ./output.mp3` |
|
||||
| Download video | `notebooklm download video ./output.mp4` |
|
||||
| Download cinematic video | `notebooklm download cinematic-video ./cinematic.mp4` (alias for `download video`) |
|
||||
| Download infographic | `notebooklm download infographic ./infographic.png` |
|
||||
| Download slide deck (PDF) | `notebooklm download slide-deck ./slides.pdf` |
|
||||
| Download slide deck (PPTX) | `notebooklm download slide-deck ./slides.pptx --format pptx` |
|
||||
| Download report | `notebooklm download report ./report.md` |
|
||||
| Download mind map | `notebooklm download mind-map ./map.json` |
|
||||
| Download data table | `notebooklm download data-table ./data.csv` |
|
||||
| Download quiz | `notebooklm download quiz quiz.json` |
|
||||
| Download quiz (markdown) | `notebooklm download quiz --format markdown quiz.md` |
|
||||
| Download flashcards | `notebooklm download flashcards cards.json` |
|
||||
| Download flashcards (markdown) | `notebooklm download flashcards --format markdown cards.md` |
|
||||
| Delete notebook | `notebooklm delete -n <id>` (add `--yes` to skip the prompt non-interactively) |
|
||||
| List languages | `notebooklm language list` |
|
||||
| Get language | `notebooklm language get` |
|
||||
| Set language | `notebooklm language set zh_Hans` |
|
||||
| List profiles | `notebooklm profile list` |
|
||||
| Create profile | `notebooklm profile create work` |
|
||||
| Switch profile | `notebooklm profile switch work` |
|
||||
| Delete profile | `notebooklm profile delete old --yes` (`-y`; `--confirm` is a deprecated alias) |
|
||||
| Rename profile | `notebooklm profile rename old new` |
|
||||
| Use profile (one-off) | `notebooklm -p work list` |
|
||||
| Health check | `notebooklm doctor` |
|
||||
| Health check (auto-fix) | `notebooklm doctor --fix` |
|
||||
|
||||
**Parallel safety:** Use explicit notebook IDs in parallel workflows. Notebook-scoped commands broadly support `-n/--notebook` (ask/history, source, artifact, generate, download, note, label, share, research, and notebook delete/rename/summary/metadata). Download commands also support `-a/--artifact`. For chat, use `-c <conversation_id>` to target a specific conversation.
|
||||
|
||||
**Partial IDs:** Use first 6+ characters of UUIDs. Must be unique prefix (fails if ambiguous). Works for ID-based commands such as `use`, `source delete`, and `wait`. For exact source-title deletion, use `source delete-by-title "Title"`. For automation, prefer full UUIDs to avoid ambiguity.
|
||||
|
||||
## Command Output Formats
|
||||
|
||||
Commands with `--json` return structured data for parsing:
|
||||
|
||||
**Create notebook:**
|
||||
```bash
|
||||
$ notebooklm create "Research" --json
|
||||
{"notebook": {"id": "abc123de-...", "title": "Research", "created_at": null}}
|
||||
# parse with: jq -r .notebook.id
|
||||
```
|
||||
|
||||
**Add source:**
|
||||
```bash
|
||||
$ notebooklm source add "https://example.com" --json
|
||||
{"source": {"id": "def456...", "title": "Example", "type": "web_page", "url": "https://example.com"}}
|
||||
# parse with: jq -r .source.id
|
||||
# Note: no `status` field on add — use `source list --json` or `source wait` to check processing state.
|
||||
```
|
||||
|
||||
**Generate artifact:**
|
||||
```bash
|
||||
$ notebooklm generate audio "Focus on key points" --json
|
||||
{"task_id": "xyz789...", "status": "pending"}
|
||||
# When run with --wait, completed status also includes a `url` field.
|
||||
```
|
||||
|
||||
**Chat with references:**
|
||||
```bash
|
||||
$ notebooklm ask "What is X?" --json
|
||||
{"answer": "X is... [1] [2]", "conversation_id": "...", "turn_number": 1, "is_follow_up": false, "references": [{"source_id": "abc123...", "citation_number": 1, "cited_text": "Relevant passage from source..."}, {"source_id": "def456...", "citation_number": 2, "cited_text": "Another passage..."}]}
|
||||
```
|
||||
|
||||
**Source fulltext (get indexed content):**
|
||||
```bash
|
||||
$ notebooklm source fulltext <source_id> --json
|
||||
{"source_id": "...", "title": "...", "kind": "web_page", "content": "Full indexed text...", "url": null, "char_count": 12345}
|
||||
```
|
||||
|
||||
**Understanding citations:** The `cited_text` in references is often a snippet or section header, not the full quoted passage. The `start_char`/`end_char` positions reference NotebookLM's internal chunked index, not the raw fulltext. Use `SourceFulltext.find_citation_context()` to locate citations:
|
||||
```python
|
||||
fulltext = await client.sources.get_fulltext(notebook_id, ref.source_id)
|
||||
matches = fulltext.find_citation_context(ref.cited_text) # Returns list[(context, position)]
|
||||
if matches:
|
||||
context, pos = matches[0] # First match; check len(matches) > 1 for duplicates
|
||||
```
|
||||
|
||||
**Extract IDs:** Singular endpoints wrap their result in an envelope —
|
||||
parse `.notebook.id` (from `create`), `.source.id` (from `source add`),
|
||||
or `.task_id` (from `generate *`). The chat `--json` references list uses
|
||||
`.references[].source_id`.
|
||||
|
||||
## Generation Types
|
||||
|
||||
Common generate options vary by subcommand:
|
||||
- `-n, --notebook` targets the notebook.
|
||||
- `-s, --source` limits generation to specific source(s) on content generators (not `revise-slide`).
|
||||
- `--language` sets output language where supported (defaults to configured language or `en`).
|
||||
- `--wait`, `--timeout`, and `--interval` are shared polling controls where waiting is supported.
|
||||
- `--json` returns machine-readable output.
|
||||
- `--retry N` automatically retries rate limits on supported subcommands (not `mind-map`).
|
||||
- `--prompt-file PATH` reads description/query text from a file on `ask`, generation subcommands except `mind-map`, and `source add-research`.
|
||||
|
||||
| Type | Command | Options | Download |
|
||||
|------|---------|---------|----------|
|
||||
| Podcast | `generate audio` | `--format [deep-dive\|brief\|critique\|debate]`, `--length [short\|default\|long]` | .mp3 |
|
||||
| Video | `generate video` | `--format [explainer\|brief\|cinematic\|short]` (⁴), `--style [auto\|custom\|classic\|whiteboard\|kawaii\|anime\|watercolor\|retro-print\|heritage\|paper-craft]`, `--style-prompt` with `--style custom` | .mp4 |
|
||||
| Slide Deck | `generate slide-deck` | `--format [detailed\|presenter]`, `--length [default\|short]` (²) | .pdf / .pptx |
|
||||
| Slide Revision | `generate revise-slide "prompt" --artifact <id> --slide N` | `--wait`, `--notebook` | *(re-downloads parent deck)* |
|
||||
| Infographic | `generate infographic` | `--orientation [landscape\|portrait\|square]`, `--detail [concise\|standard\|detailed]`, `--style [auto\|sketch-note\|professional\|bento-grid\|editorial\|instructional\|bricks\|clay\|anime\|kawaii\|scientific]` | .png |
|
||||
| Report | `generate report` | `--format [briefing-doc\|study-guide\|blog-post\|custom]`, `--append "extra instructions"` (¹) | .md |
|
||||
| Mind Map | `generate mind-map` | `--kind [interactive\|note-backed]` (³) *(default: interactive)* | .json |
|
||||
| Data Table | `generate data-table` | description required | .csv |
|
||||
| Quiz | `generate quiz` | `--difficulty [easy\|medium\|hard]`, `--quantity [fewer\|standard\|more]` | .json/.md/.html |
|
||||
| Flashcards | `generate flashcards` | `--difficulty [easy\|medium\|hard]`, `--quantity [fewer\|standard\|more]` | .json/.md/.html |
|
||||
|
||||
¹ `--append` only customizes the built-in templates. With `--format custom`, pass the prompt as the positional `DESCRIPTION` argument (`notebooklm generate report "PROMPT" --format custom`); `--append` is silently ignored in that mode (the CLI prints a warning).
|
||||
|
||||
³ **Two kinds of mind map (issue #1256).** `generate mind-map --kind interactive` (the default) creates the **interactive** studio artifact (what the web app now makes); it is polled to completion. `generate mind-map --kind note-backed` creates the **note-backed** kind — a JSON node tree, generated synchronously. Both emit the same `{mind_map, note_id, kind}` JSON, list under `artifact list --type mind-map`, and export via `download mind-map`. `--instructions` applies only to the note-backed kind.
|
||||
|
||||
⁴ **Cinematic video (Veo 3).** `generate video --format cinematic` generates AI documentary footage via Veo 3; it **ignores `--style`**, takes ~30-40 min, and requires a Google AI Ultra subscription. Also exposed as the `generate cinematic-video` alias (which forces `--format cinematic` and a longer default timeout). Download with `download video` or the `download cinematic-video` alias.
|
||||
|
||||
² **Portrait / vertical slide decks via prompt.** Slide-deck has no `--orientation` flag (unlike infographic). Treat portrait decks as skill-level prompt guidance, not a typed CLI/API contract: NotebookLM currently honors orientation cues written into the `DESCRIPTION` positional argument. Including phrases like `"9:16 portrait"`, `"vertical layout"`, `"portrait mobile format"`, or `"vertical 9:16 layout"` can make NotebookLM render each slide as a 9:16 portrait image. Empirically:
|
||||
|
||||
- The `.pptx` canvas itself may stay 16:9, but each slide's embedded image can be rendered as 9:16 portrait — useful for vertical/mobile video material extracted via `python-pptx`.
|
||||
- Orientation is steered once at generation time. `generate revise-slide` edits content within an existing slide but does not change its orientation; if a slide falls back to landscape (occasional inconsistency), regenerate the whole deck rather than revising the single page.
|
||||
- Combine with an explicit page count in the prompt (e.g. `"Create exactly 8 pages, using a vertical 9:16 portrait layout"`) for the most predictable output.
|
||||
|
||||
```bash
|
||||
# Skill prompt hint: ask NotebookLM to render each slide as a 9:16 portrait image
|
||||
notebooklm generate slide-deck "Create an 8-page deck in 9:16 portrait orientation for mobile viewing" --length default
|
||||
```
|
||||
|
||||
## Features Beyond the Web UI
|
||||
|
||||
These capabilities are available via CLI but not in NotebookLM's web interface:
|
||||
|
||||
| Feature | Command | Description |
|
||||
|---------|---------|-------------|
|
||||
| **Batch downloads** | `download <type> --all` | Download all artifacts of a type at once |
|
||||
| **Quiz/Flashcard export** | `download quiz --format json` | Export as JSON, Markdown, or HTML (web UI only shows interactive view) |
|
||||
| **Mind map extraction** | `download mind-map` | Export hierarchical JSON for visualization tools |
|
||||
| **Data table export** | `download data-table` | Download structured tables as CSV |
|
||||
| **Slide deck as PPTX** | `download slide-deck --format pptx` | Download slide deck as editable .pptx (web UI only offers PDF) |
|
||||
| **Slide revision** | `generate revise-slide "prompt" --artifact <id> --slide N` | Modify individual slides with a natural-language prompt |
|
||||
| **Report template append** | `generate report --format study-guide --append "..."` | Append custom instructions to built-in format templates without losing the format type |
|
||||
| **Source fulltext** | `source fulltext <id>` | Retrieve the indexed text content of any source |
|
||||
| **Save chat to note** | `ask "..." --save-as-note` / `history --save` | Save Q&A answers or conversation history as notebook notes |
|
||||
| **Programmatic sharing** | `share` commands | Manage sharing permissions without the UI |
|
||||
|
||||
## Common Workflows
|
||||
|
||||
### Research to Podcast (Interactive)
|
||||
**Time:** 5-10 minutes total
|
||||
|
||||
1. `notebooklm create "Research: [topic]"` — *if fails: check auth with `notebooklm login`*
|
||||
2. `notebooklm source add` for each URL/document — *if one fails: log warning, continue with others*
|
||||
3. Wait for sources: `notebooklm source list --json` until all status=READY — *required before generation*
|
||||
4. `notebooklm generate audio "Focus on [specific angle]"` (confirm when asked) — *if rate limited: wait 5 min, retry once*
|
||||
5. Note the artifact ID returned
|
||||
6. Check `notebooklm artifact list` later for status
|
||||
7. `notebooklm download audio ./podcast.mp3` when complete (confirm when asked)
|
||||
|
||||
### Research to Podcast (Automated with Subagent)
|
||||
**Time:** 5-10 minutes, but continues in background
|
||||
|
||||
When user wants full automation (generate and download when ready):
|
||||
|
||||
1. Create notebook and add sources as usual
|
||||
2. Wait for sources to be ready (use `source wait` or check `source list --json`)
|
||||
3. Run `notebooklm generate audio "..." --json` → parse `task_id` from output
|
||||
4. **Spawn a background agent** using Task tool:
|
||||
```python
|
||||
Task(
|
||||
prompt="Wait for artifact {task_id} in notebook {notebook_id} to complete, then download.
|
||||
Use: notebooklm artifact wait {task_id} -n {notebook_id} --timeout 1200
|
||||
Then: notebooklm download audio ./podcast.mp3 -a {task_id} -n {notebook_id}",
|
||||
subagent_type="general-purpose"
|
||||
)
|
||||
```
|
||||
5. Main conversation continues while agent waits
|
||||
|
||||
**Error handling in subagent:**
|
||||
- If `artifact wait` returns exit code 2 (timeout): Report timeout, suggest checking `artifact list`
|
||||
- If download fails: Check if artifact status is COMPLETED first
|
||||
|
||||
**Benefits:** Non-blocking, user can do other work, automatic download on completion
|
||||
|
||||
### Document Analysis
|
||||
**Time:** 1-2 minutes
|
||||
|
||||
1. `notebooklm create "Analysis: [project]"`
|
||||
2. `notebooklm source add ./doc.pdf` (or URLs)
|
||||
3. `notebooklm ask "Summarize the key points"`
|
||||
4. `notebooklm ask "What are the main arguments?"`
|
||||
5. Continue chatting as needed
|
||||
|
||||
### Bulk Import
|
||||
**Time:** Varies by source count
|
||||
|
||||
1. `notebooklm create "Collection: [name]"`
|
||||
2. Add multiple sources:
|
||||
```bash
|
||||
notebooklm source add "https://url1.com"
|
||||
notebooklm source add "https://url2.com"
|
||||
notebooklm source add ./local-file.pdf
|
||||
```
|
||||
3. `notebooklm source list` to verify
|
||||
|
||||
**Source limits:** Varies by plan—Standard: 50, Plus: 100, Pro: 300, Ultra: 600 sources per notebook. See [NotebookLM plans](https://support.google.com/notebooklm/answer/16213268) for details. The CLI does not enforce these limits; they are applied by your NotebookLM account.
|
||||
**Supported types:** PDFs, YouTube URLs, web URLs, Google Docs, text files, Markdown, Word docs, EPUB, audio files, video files, images
|
||||
|
||||
### Bulk Import with Source Waiting (Subagent Pattern)
|
||||
**Time:** Varies by source count
|
||||
|
||||
When adding multiple sources and needing to wait for processing before chat/generation:
|
||||
|
||||
1. Add sources with `--json` to capture IDs (parse with `jq -r .source.id`):
|
||||
```bash
|
||||
notebooklm source add "https://url1.com" --json # → {"source": {"id": "abc...", ...}}
|
||||
notebooklm source add "https://url2.com" --json # → {"source": {"id": "def...", ...}}
|
||||
```
|
||||
2. **Spawn a background agent** to wait for all sources:
|
||||
```
|
||||
Task(
|
||||
prompt="Wait for sources {source_ids} in notebook {notebook_id} to be ready.
|
||||
For each: notebooklm source wait {id} -n {notebook_id} --timeout 600
|
||||
Report when all ready or if any fail.",
|
||||
subagent_type="general-purpose"
|
||||
)
|
||||
```
|
||||
3. Main conversation continues while agent waits
|
||||
4. Once sources are ready, proceed with chat or generation
|
||||
|
||||
**Why wait for sources?** Sources must be indexed before chat or generation. Takes ~30 seconds to several minutes per source (see the processing-times table below).
|
||||
|
||||
### Deep Web Research (Subagent Pattern)
|
||||
**Time:** 15-30+ minutes, runs in background
|
||||
|
||||
Deep research finds and analyzes web sources on a topic:
|
||||
|
||||
1. Create notebook: `notebooklm create "Research: [topic]"`
|
||||
2. Start deep research (non-blocking):
|
||||
```bash
|
||||
notebooklm source add-research "topic query" --mode deep --no-wait
|
||||
```
|
||||
3. **Spawn a background agent** to wait and import:
|
||||
```
|
||||
Task(
|
||||
prompt="Wait for research in notebook {notebook_id} to complete and import sources.
|
||||
Use: notebooklm research wait -n {notebook_id} --import-all --timeout 1800
|
||||
Report how many sources were imported.",
|
||||
subagent_type="general-purpose"
|
||||
)
|
||||
```
|
||||
4. Main conversation continues while agent waits
|
||||
5. When agent completes, sources are imported automatically
|
||||
|
||||
**Alternative (blocking):** For simple cases, omit `--no-wait`:
|
||||
```bash
|
||||
notebooklm source add-research "topic" --mode deep --import-all
|
||||
# Blocks until research completes (deep mode: 15-30+ min)
|
||||
```
|
||||
|
||||
**When to use each mode:**
|
||||
- `--mode fast`: Specific topic, quick overview needed (5-10 sources, seconds)
|
||||
- `--mode deep`: Broad topic, comprehensive analysis needed (20+ sources, 15-30+ min)
|
||||
|
||||
**Research sources:**
|
||||
- `--from web`: Search the web (default)
|
||||
- `--from drive`: Search Google Drive
|
||||
|
||||
## Output Style
|
||||
|
||||
**Progress updates:** Brief status for each step
|
||||
- "Creating notebook 'Research: AI'..."
|
||||
- "Adding source: https://example.com..."
|
||||
- "Starting audio generation... (task ID: abc123)"
|
||||
|
||||
**Fire-and-forget for long operations:**
|
||||
- Start generation, return artifact ID immediately
|
||||
- Do NOT poll or wait in main conversation - generation takes 5-45 minutes (see timing table)
|
||||
- User checks status manually, OR use subagent with `artifact wait`
|
||||
|
||||
**JSON output:** Use `--json` flag for machine-readable output:
|
||||
```bash
|
||||
notebooklm list --json
|
||||
notebooklm auth check --test --json # use --test for network-validated auth (see § Agent Setup Verification)
|
||||
notebooklm source list --json
|
||||
notebooklm artifact list --json
|
||||
```
|
||||
|
||||
**JSON schemas (key fields):**
|
||||
|
||||
`notebooklm list --json`:
|
||||
```json
|
||||
{"notebooks": [{"index": 1, "id": "...", "title": "...", "is_owner": true, "created_at": "..."}], "count": 1}
|
||||
```
|
||||
|
||||
`notebooklm auth check --test --json` (use `--test` to drive the network token-fetch — bare `--json` would leave `"token_fetch": null`):
|
||||
```json
|
||||
{"status": "ok", "checks": {"storage_exists": true, "json_valid": true, "cookies_present": true, "sid_cookie": true, "token_fetch": true}, "details": {"storage_path": "...", "auth_source": "file", "cookies_found": ["SID", "HSID", "..."], "cookie_domains": [".google.com"]}}
|
||||
```
|
||||
|
||||
`notebooklm source list --json`:
|
||||
```json
|
||||
{"notebook_id": "...", "notebook_title": "...", "sources": [{"index": 1, "id": "...", "title": "...", "type": "web_page", "url": "...", "status": "ready|processing|error", "status_id": 1, "created_at": "..."}], "count": 1}
|
||||
```
|
||||
|
||||
`notebooklm artifact list --json`:
|
||||
```json
|
||||
{"notebook_id": "...", "notebook_title": "...", "artifacts": [{"index": 1, "id": "...", "title": "...", "type": "Audio", "type_id": 1, "status": "in_progress|pending|completed|unknown", "status_id": 1, "created_at": "..."}], "count": 1}
|
||||
```
|
||||
|
||||
**Status values:**
|
||||
- Sources: `processing` → `ready` (or `error`)
|
||||
- Artifacts: `pending` or `in_progress` → `completed` (or `unknown`)
|
||||
|
||||
## Error Handling
|
||||
|
||||
**On failure, offer the user a choice:**
|
||||
1. Retry the operation
|
||||
2. Skip and continue with something else
|
||||
3. Investigate the error
|
||||
|
||||
**Error decision tree:**
|
||||
|
||||
| Error | Cause | Action |
|
||||
|-------|-------|--------|
|
||||
| Auth/cookie error | Session expired | Run `notebooklm auth check` then `notebooklm login` |
|
||||
| "No notebook context" | Context not set | Use `-n <id>` or `--notebook <id>` flag (parallel), or `notebooklm use <id>` (single-agent) |
|
||||
| "No result found for RPC ID" | Rate limiting | Wait 5-10 min, retry |
|
||||
| `GENERATION_FAILED` | Google rate limit | Wait and retry later |
|
||||
| Download fails | Generation incomplete | Check `artifact list` for status |
|
||||
| Invalid notebook/source ID | Wrong ID | Run `notebooklm list` to verify |
|
||||
| RPC protocol error | Google changed APIs | May need CLI update |
|
||||
|
||||
## Exit Codes
|
||||
|
||||
All commands use consistent exit codes:
|
||||
|
||||
| Code | Meaning | Action |
|
||||
|------|---------|--------|
|
||||
| 0 | Success | Continue |
|
||||
| 1 | Error (not found, processing failed) | Check stderr, see Error Handling |
|
||||
| 2 | Timeout (wait commands only) | Extend timeout or check status manually |
|
||||
|
||||
**Examples:**
|
||||
- `source wait` returns 1 if source not found or processing failed
|
||||
- `artifact wait` returns 2 if timeout reached before completion
|
||||
- `generate` returns 1 if rate limited (check stderr for details)
|
||||
|
||||
## Long Prompts
|
||||
|
||||
When a prompt or query exceeds shell command-line length limits, use `--prompt-file` to read it from a file:
|
||||
|
||||
```bash
|
||||
notebooklm ask --prompt-file ./long_question.txt
|
||||
notebooklm generate report --prompt-file ./custom_report_prompt.txt
|
||||
notebooklm source add-research --prompt-file ./research_query.txt --mode deep
|
||||
```
|
||||
|
||||
`--prompt-file` is mutually exclusive with the positional text argument. The file is read as UTF-8 with trailing whitespace stripped. Supported on: `ask`, all `generate` subcommands (except `mind-map`), and `source add-research`.
|
||||
|
||||
> **Note:** `--prompt-file` reads a *prompt/query text file*, not a source document. To upload a file as a notebook source, use `source add ./file.pdf`.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
**Rate limiting:** Audio, video, quiz, flashcards, infographic, and slide deck generation may fail due to Google's rate limits. This is an API limitation, not a bug.
|
||||
|
||||
**Reliable operations:** These always work:
|
||||
- Notebooks (list, create, delete, rename)
|
||||
- Sources (add, list, delete)
|
||||
- Chat/queries
|
||||
- Mind-map, study-guide, report, data-table generation
|
||||
|
||||
**Unreliable operations:** These may fail with rate limiting:
|
||||
- Audio (podcast) generation
|
||||
- Video generation
|
||||
- Quiz and flashcard generation
|
||||
- Infographic and slide deck generation
|
||||
|
||||
**Workaround:** If generation fails:
|
||||
1. Check status: `notebooklm artifact list`
|
||||
2. Retry after 5-10 minutes
|
||||
3. Use the NotebookLM web UI as fallback
|
||||
|
||||
**Processing times vary significantly.** Use the subagent pattern for long operations:
|
||||
|
||||
| Operation | Typical time | Suggested timeout |
|
||||
|-----------|--------------|-------------------|
|
||||
| Source processing | 30s - 10 min | 600s |
|
||||
| Research (fast) | 30s - 2 min | 180s |
|
||||
| Research (deep) | 15 - 30+ min | 1800s |
|
||||
| Notes | instant | n/a |
|
||||
| Mind-map | instant (sync) | n/a |
|
||||
| Quiz, flashcards | 5 - 15 min | 900s |
|
||||
| Report, data-table | 5 - 15 min | 900s |
|
||||
| Audio generation | 10 - 20 min | 1200s |
|
||||
| Video generation | 15 - 45 min | 2700s |
|
||||
|
||||
**Polling intervals:** When checking status manually, poll every 15-30 seconds to avoid excessive API calls.
|
||||
|
||||
## Language Configuration
|
||||
|
||||
Language setting controls the output language for generated artifacts (audio, video, etc.).
|
||||
|
||||
**Important:** Language is a **GLOBAL** setting that affects all notebooks in your account.
|
||||
|
||||
```bash
|
||||
# List all 80+ supported languages with native names
|
||||
notebooklm language list
|
||||
|
||||
# Show current language setting
|
||||
notebooklm language get
|
||||
|
||||
# Set language for artifact generation
|
||||
notebooklm language set zh_Hans # Simplified Chinese
|
||||
notebooklm language set ja # Japanese
|
||||
notebooklm language set en # English (default)
|
||||
```
|
||||
|
||||
**Common language codes:**
|
||||
| Code | Language |
|
||||
|------|----------|
|
||||
| `en` | English |
|
||||
| `zh_Hans` | 中文(简体) - Simplified Chinese |
|
||||
| `zh_Hant` | 中文(繁體) - Traditional Chinese |
|
||||
| `ja` | 日本語 - Japanese |
|
||||
| `ko` | 한국어 - Korean |
|
||||
| `es` | Español - Spanish |
|
||||
| `fr` | Français - French |
|
||||
| `de` | Deutsch - German |
|
||||
| `pt_BR` | Português (Brasil) |
|
||||
|
||||
**Override per command:** Use `--language` flag on generate commands:
|
||||
```bash
|
||||
notebooklm generate audio --language ja # Japanese podcast
|
||||
notebooklm generate video --language zh_Hans # Chinese video
|
||||
```
|
||||
|
||||
**Offline mode:** Use `--local` flag to skip server sync:
|
||||
```bash
|
||||
notebooklm language set zh_Hans --local # Save locally only
|
||||
notebooklm language get --local # Read local config only
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
```bash
|
||||
notebooklm --help # Main commands
|
||||
notebooklm auth check # Diagnose auth issues
|
||||
notebooklm auth check --test # Full auth validation with network test
|
||||
notebooklm source --help # Source management
|
||||
notebooklm research --help # Research status/wait/cancel
|
||||
notebooklm generate --help # Content generation
|
||||
notebooklm artifact --help # Artifact management
|
||||
notebooklm download --help # Download content
|
||||
notebooklm language --help # Language settings
|
||||
```
|
||||
|
||||
**Diagnose auth:** `notebooklm auth check` - shows cookie domains, storage path, validation status
|
||||
**Re-authenticate:** `notebooklm login`
|
||||
**Check version:** `notebooklm --version`
|
||||
**Refresh a CLI-managed install:** `notebooklm skill install`
|
||||
@@ -0,0 +1,67 @@
|
||||
# Copy to .env (which is gitignored) and fill in. NEVER commit real values.
|
||||
# Or just run `make setup` — it picks a tunnel, generates the secrets below, and
|
||||
# writes this file for you.
|
||||
|
||||
# Which tunnel to run (Compose profile): cloudflare or tailscale. `make setup`
|
||||
# writes this; `make up`/`make dev` read it (a `TUNNEL=... make up` override wins).
|
||||
TUNNEL=cloudflare
|
||||
|
||||
# Which published image `make up`/`make prod` pull. By default, make uses this
|
||||
# checkout's pyproject version. Set this only for raw docker compose, a fork, or
|
||||
# a pinned tag.
|
||||
# NOTEBOOKLM_MCP_IMAGE=tenglin/notebooklm-mcp
|
||||
# NOTEBOOKLM_MCP_VERSION=<version>
|
||||
|
||||
# Bearer token clients must present to use the MCP endpoint. Generate a strong
|
||||
# random value, e.g.: python -c "import secrets; print(secrets.token_urlsafe(32))"
|
||||
# Works with Claude Code / Desktop (which send an Authorization header).
|
||||
NOTEBOOKLM_MCP_TOKEN=
|
||||
|
||||
# --- Optional: self-hosted OAuth (only needed to connect from claude.ai, whose ---
|
||||
# connector UI is OAuth-only). Leave BOTH unset to stay bearer-only (Claude
|
||||
# Code/Desktop unaffected). When set, they're additive — the bearer keeps working
|
||||
# AND claude.ai can connect via OAuth, gated by this one password. No external IdP.
|
||||
# - PASSWORD: the gate. Use a long random value (>=16 chars); it's the primary
|
||||
# brute-force defense. Generate: python -c "import secrets; print(secrets.token_urlsafe(24))"
|
||||
# - BASE_URL: the public https URL claude.ai reaches (your tunnel hostname).
|
||||
# NOTE: Cloudflare's edge sees this password in transit (it terminates TLS) — use a
|
||||
# throwaway Google account. Rotating the password swaps the in-memory check but does
|
||||
# NOT revoke already-issued OAuth tokens (they're long-lived and persisted): real
|
||||
# revocation = delete the profile's oauth_state.json and restart. Both vars required
|
||||
# together (partial → refuses to start).
|
||||
# NOTEBOOKLM_MCP_OAUTH_PASSWORD=
|
||||
# NOTEBOOKLM_MCP_OAUTH_BASE_URL=https://your-host.example.com
|
||||
# Cloudflare ONLY: trust the tunnel's CF-Connecting-IP header so the login throttle keys
|
||||
# per-IP instead of globally. Leave unset unless a trusted proxy sets that header (an
|
||||
# exposed-directly origin could forge it to dodge the throttle). Tailscale Funnel does not
|
||||
# set it, so this is a no-op there.
|
||||
# NOTEBOOKLM_MCP_TRUST_PROXY=1
|
||||
|
||||
# --- Tunnel: pick ONE (see README §"Connect from claude.ai" / the tunnel options) ---
|
||||
# A) Cloudflare (`make dev` / TUNNEL=cloudflare) — needs a domain in your Cloudflare
|
||||
# account. Token: Cloudflare dashboard → Networking → Tunnels → your tunnel → its token.
|
||||
# Route: Routes → Add route → Published application, Service URL http://notebooklm-mcp:9420.
|
||||
CF_TUNNEL_TOKEN=
|
||||
# Optional: pin the cloudflared image to a release from
|
||||
# github.com/cloudflare/cloudflared/releases (defaults to `latest`).
|
||||
# CLOUDFLARED_VERSION=2025.1.0
|
||||
|
||||
# B) Tailscale Funnel (`make dev TUNNEL=tailscale`) — NO domain; free stable HTTPS at
|
||||
# https://notebooklm-mcp.<your-tailnet>.ts.net. One-time in the Tailscale admin
|
||||
# console: enable MagicDNS + HTTPS certificates, and grant the `funnel` node
|
||||
# attribute via Settings → Funnel → Manage (deep-links to the ACL policy editor;
|
||||
# add nodeAttrs {"target":["*"],"attr":["funnel"]} and Save). TS_AUTHKEY is a
|
||||
# NORMAL auth key (Settings → Keys) — Funnel comes from the policy, not the key.
|
||||
# Then set NOTEBOOKLM_MCP_OAUTH_BASE_URL to that ts.net origin (tailscale profile only).
|
||||
# TS_AUTHKEY=tskey-auth-xxxxxxxx
|
||||
|
||||
# Which host profile dir to mount — the Google account the server drives.
|
||||
# Defaults to your local "default" profile. Point at a dedicated/throwaway
|
||||
# profile here (recommended — master_token.json is a full-account credential).
|
||||
# NOTEBOOKLM_PROFILE_DIR=/home/you/.notebooklm/profiles/throwaway
|
||||
|
||||
# The container runs as this uid:gid so the mounted profile needs NO chown (your
|
||||
# own CLI keeps owning the files). `make` fills these from `id` automatically;
|
||||
# set them only for raw `docker compose` (use your own: id -u / id -g).
|
||||
# NOTEBOOKLM_UID=1000
|
||||
# NOTEBOOKLM_GID=1000
|
||||
@@ -0,0 +1,3 @@
|
||||
# Never commit deployment secrets.
|
||||
.env
|
||||
profile/
|
||||
@@ -0,0 +1,50 @@
|
||||
# Remote notebooklm-mcp server — single-tenant, bearer-gated, streamable-HTTP.
|
||||
#
|
||||
# The image is stateless: the profile (incl. master_token.json) is mounted at
|
||||
# runtime, never baked in. Run behind a Cloudflare Tunnel (or any TLS reverse
|
||||
# proxy) — see deploy/README.md.
|
||||
#
|
||||
# Two install modes (the compose build context is the repo root):
|
||||
# * DEV / from-source (DEFAULT): installs THIS checkout, so `docker compose up`
|
||||
# deploys exactly the code in this repo. Leave NOTEBOOKLM_SPEC empty.
|
||||
# * PRODUCTION / from PyPI: build with a release spec, e.g.
|
||||
# docker compose build --build-arg NOTEBOOKLM_SPEC="notebooklm-py[mcp,headless]==<version>"
|
||||
# (or set it under build.args in docker-compose.yml).
|
||||
FROM python:3.12-slim
|
||||
|
||||
ARG NOTEBOOKLM_SPEC=""
|
||||
|
||||
# Copy the repo source (build context = repo root; see .dockerignore for excludes).
|
||||
COPY . /src
|
||||
|
||||
# Empty NOTEBOOKLM_SPEC → install the copied source (dev). Non-empty → install
|
||||
# that spec from PyPI (production) and ignore the copied source.
|
||||
RUN if [ -n "$NOTEBOOKLM_SPEC" ]; then \
|
||||
pip install --no-cache-dir "$NOTEBOOKLM_SPEC"; \
|
||||
else \
|
||||
pip install --no-cache-dir "/src[mcp,headless]"; \
|
||||
fi
|
||||
|
||||
# Run as a non-root user; the mounted profile dir must be writable by this uid
|
||||
# (keepalive/recovery rotates cookies into storage_state.json).
|
||||
RUN useradd --create-home --uid 10001 app
|
||||
USER app
|
||||
|
||||
ENV NOTEBOOKLM_MCP_HOST=0.0.0.0 \
|
||||
NOTEBOOKLM_MCP_PORT=9420 \
|
||||
NOTEBOOKLM_PROFILE=server
|
||||
|
||||
EXPOSE 9420
|
||||
|
||||
# Liveness: stdlib-only TCP connect to the listener over loopback. No curl/wget/nc
|
||||
# needed, and it never touches the auth-gated /mcp route. The python process reads
|
||||
# NOTEBOOKLM_MCP_PORT from its own env at runtime (a program sees its environment
|
||||
# even in exec-form — only shell `$VAR` substitution doesn't), so overriding the
|
||||
# port keeps the healthcheck in sync. Host is always loopback: the server binds
|
||||
# 0.0.0.0, which you connect to via 127.0.0.1 from inside the container.
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
|
||||
CMD ["python", "-c", "import os,socket,sys; p=int(os.environ.get('NOTEBOOKLM_MCP_PORT','9420')); s=socket.socket(); s.settimeout(2); sys.exit(0 if s.connect_ex(('127.0.0.1',p))==0 else 1)"]
|
||||
|
||||
# --host 0.0.0.0 is non-loopback, so NOTEBOOKLM_MCP_ALLOW_EXTERNAL_BIND=1 AND a
|
||||
# NOTEBOOKLM_MCP_TOKEN are MANDATORY (the entrypoint refuses to start otherwise).
|
||||
CMD ["notebooklm-mcp", "--transport", "http"]
|
||||
+100
@@ -0,0 +1,100 @@
|
||||
# Convenience wrapper for the remote notebooklm-mcp deploy.
|
||||
# make setup — interactive first run: pick a tunnel + generate secrets → .env
|
||||
# make up — pull the published image and start (the easy path)
|
||||
# make prod VERSION=x.y.z — pull a specific published version and start
|
||||
# make dev — build THIS checkout + start (contributors)
|
||||
# make dev TUNNEL=tailscale — same, but with the Tailscale Funnel sidecar (no domain)
|
||||
# make logs / restart / down / config
|
||||
# TUNNEL selects the tunnel sidecar (Compose profile): cloudflare (default) or tailscale.
|
||||
# Override the compose command with COMPOSE=... if needed.
|
||||
|
||||
COMPOSE ?= docker compose
|
||||
# TUNNEL: a CLI/env override wins; else the choice `make setup` saved in .env;
|
||||
# else cloudflare. Only TUNNEL is read from .env into make — it's make-only (not a
|
||||
# compose var), so it can't collide with compose's own reading of .env.
|
||||
_TUNNEL_DOTENV := $(shell [ -f .env ] && sed -n 's/^TUNNEL=//p' .env | tr -d '\r' | head -1)
|
||||
TUNNEL ?= $(if $(_TUNNEL_DOTENV),$(_TUNNEL_DOTENV),cloudflare)
|
||||
|
||||
# Pin a published image tag with `make prod VERSION=x.y.z`. Otherwise, keep the
|
||||
# override order: shell/CLI env, deploy/.env, this checkout's pyproject version,
|
||||
# then Make's missing-version guard.
|
||||
_PROJECT_VERSION := $(shell [ -f ../pyproject.toml ] && \
|
||||
sed -n '/^\[project\]/,/^\[/ s/^[[:space:]]*version[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' ../pyproject.toml | tr -d '\r' | head -1)
|
||||
_VERSION_DOTENV := $(shell [ -f .env ] && sed -n 's/^NOTEBOOKLM_MCP_VERSION=//p' .env | tr -d '\r' | head -1)
|
||||
ifeq ($(origin VERSION),command line)
|
||||
ifneq ($(strip $(VERSION)),)
|
||||
COMPOSE_ENV := NOTEBOOKLM_MCP_VERSION=$(VERSION)
|
||||
endif
|
||||
else ifneq ($(origin NOTEBOOKLM_MCP_VERSION),undefined)
|
||||
COMPOSE_ENV :=
|
||||
else ifneq ($(_VERSION_DOTENV),)
|
||||
COMPOSE_ENV :=
|
||||
else ifneq ($(_PROJECT_VERSION),)
|
||||
COMPOSE_ENV := NOTEBOOKLM_MCP_VERSION=$(_PROJECT_VERSION)
|
||||
endif
|
||||
|
||||
# Profile-aware compose invocation — selects exactly one tunnel sidecar.
|
||||
DC = $(COMPOSE_ENV) $(COMPOSE) --profile $(TUNNEL)
|
||||
# Same, but layer the build override so the image is built from THIS checkout.
|
||||
DC_BUILD = $(DC) -f docker-compose.yml -f docker-compose.build.yml
|
||||
|
||||
# Run the container as YOUR uid/gid so the mounted profile needs no chown.
|
||||
export NOTEBOOKLM_UID ?= $(shell id -u)
|
||||
export NOTEBOOKLM_GID ?= $(shell id -g)
|
||||
# NOTE: do NOT set/export NOTEBOOKLM_PROFILE_DIR here. make does not read .env, so a
|
||||
# `?=` default would resolve to "default" and `export` would push it into compose's
|
||||
# env, where it OUTRANKS the .env value (shell env > .env) — silently ignoring the
|
||||
# user's `.env` setting. Compose reads NOTEBOOKLM_PROFILE_DIR from .env directly and
|
||||
# defaults to ~/.notebooklm/profiles/default on its own; a shell/CLI override such as
|
||||
# `NOTEBOOKLM_PROFILE_DIR=… make dev` is still auto-exported by make to the recipe.
|
||||
|
||||
.PHONY: help setup dev prod up restart logs down config _require-image-version _require-tunnel-token
|
||||
|
||||
help: ## Show this help
|
||||
@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) \
|
||||
| awk 'BEGIN{FS=":.*?## "}{printf " \033[36m%-9s\033[0m %s\n", $$1, $$2}'
|
||||
|
||||
# Friendly pre-check for source-less deploys: without pyproject.toml, the user
|
||||
# must say which published image tag to pull.
|
||||
_require-image-version:
|
||||
@if [ -z "$(COMPOSE_ENV)" ] && [ -z "$$NOTEBOOKLM_MCP_VERSION" ] \
|
||||
&& ! grep -qsE '^NOTEBOOKLM_MCP_VERSION=.+' .env; then \
|
||||
echo "✗ NOTEBOOKLM_MCP_VERSION is unknown — run from a source checkout, pass VERSION=x.y.z, or set it in deploy/.env"; \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
# Friendly pre-check for the chosen tunnel's token (shell env OR deploy/.env). Compose
|
||||
# can't do a profile-scoped required-var (it interpolates the whole file before profile
|
||||
# filtering), so the guard lives here instead of a `:?` in docker-compose.yml.
|
||||
_require-tunnel-token:
|
||||
@if [ "$(TUNNEL)" = "cloudflare" ]; then \
|
||||
{ [ -n "$$CF_TUNNEL_TOKEN" ] || grep -qsE '^CF_TUNNEL_TOKEN=.+' .env; } \
|
||||
|| { echo "✗ TUNNEL=cloudflare needs CF_TUNNEL_TOKEN — set it in deploy/.env"; exit 1; }; \
|
||||
elif [ "$(TUNNEL)" = "tailscale" ]; then \
|
||||
{ [ -n "$$TS_AUTHKEY" ] || grep -qsE '^TS_AUTHKEY=.+' .env; } \
|
||||
|| { echo "✗ TUNNEL=tailscale needs TS_AUTHKEY — set it in deploy/.env"; exit 1; }; \
|
||||
else echo "✗ Unknown TUNNEL=$(TUNNEL) (use cloudflare or tailscale)"; exit 1; fi
|
||||
|
||||
setup: ## Interactive first run: pick a tunnel + generate secrets → writes .env
|
||||
@sh setup.sh
|
||||
|
||||
up: prod ## Pull the published image and start (the easy path)
|
||||
|
||||
prod: _require-image-version _require-tunnel-token ## Pull a published version and start (VERSION=x.y.z, or pin NOTEBOOKLM_MCP_VERSION in .env)
|
||||
$(DC) pull
|
||||
$(DC) up -d
|
||||
|
||||
dev: _require-image-version _require-tunnel-token ## Build THIS checkout and start (contributors; TUNNEL=cloudflare|tailscale)
|
||||
$(DC_BUILD) up -d --build
|
||||
|
||||
restart: _require-image-version _require-tunnel-token ## Rebuild THIS checkout + recreate after a source/config change
|
||||
$(DC_BUILD) up -d --build
|
||||
|
||||
logs: ## Tail the server logs
|
||||
$(COMPOSE) logs -f notebooklm-mcp
|
||||
|
||||
down: ## Stop and remove the stack (pass the same TUNNEL you started with)
|
||||
$(DC) down
|
||||
|
||||
config: ## Validate the merged compose config
|
||||
$(DC) config
|
||||
@@ -0,0 +1,274 @@
|
||||
# Remote `notebooklm-mcp` — Docker + a tunnel (Cloudflare or Tailscale)
|
||||
|
||||
Run the MCP server as a **remote connector** (Claude Code / claude.ai / Cursor)
|
||||
behind a tunnel: no public IP, no open ports, no TLS certificate to manage.
|
||||
Single-tenant, self-hosted. Pick **Cloudflare** (needs a domain) or **Tailscale
|
||||
Funnel** (no domain). `make up` pulls a **prebuilt image** — no source checkout.
|
||||
|
||||
> ⚠️ **Use a dedicated / throwaway Google account.** The mounted
|
||||
> `master_token.json` is a durable, full-account credential. Treat the mounted profile dir
|
||||
> and `.env` as secrets (both are gitignored).
|
||||
|
||||
## Quick start (the easy path)
|
||||
```bash
|
||||
# 1. bootstrap the master token once, on a machine with a browser (see §1):
|
||||
pip install "notebooklm-py[browser,headless]"
|
||||
notebooklm login --master-token --account you@example.com
|
||||
# 2. pick a tunnel + generate secrets (writes deploy/.env):
|
||||
cd deploy && make setup
|
||||
# 3. finish the tunnel setup (§3 — the one irreducible manual part), then:
|
||||
make up
|
||||
```
|
||||
`make up` pulls the published image and starts it. The rest of this doc is the
|
||||
detailed walk-through + the security model.
|
||||
|
||||
## Prerequisites
|
||||
- Docker + Docker Compose.
|
||||
- **Cloudflare tunnel:** a domain on Cloudflare (free plan is fine). **Tailscale
|
||||
Funnel:** a Tailscale account (no domain needed).
|
||||
|
||||
## 1. Bootstrap the master token (once, on a machine with a browser)
|
||||
```bash
|
||||
pip install "notebooklm-py[browser,headless]"
|
||||
notebooklm login --master-token --account you@example.com
|
||||
```
|
||||
This writes `master_token.json` (+ a minted `storage_state.json`) into
|
||||
`~/.notebooklm/profiles/<profile>/`. **You don't copy or chown anything** — the
|
||||
container mounts that dir directly and runs as *your* uid:gid, so the files stay
|
||||
owned by you (your `notebooklm` CLI keeps working) and are readable/writable with
|
||||
no permission dance.
|
||||
|
||||
- **Default:** mounts `~/.notebooklm/profiles/default`.
|
||||
- **Other profile:** set `NOTEBOOKLM_PROFILE_DIR` in `.env` (e.g. a
|
||||
dedicated/throwaway profile — recommended, since `master_token.json` is a
|
||||
full-account credential).
|
||||
|
||||
The dir is mounted **read-write** because the server re-mints/rotates cookies into
|
||||
`storage_state.json` (+ its `.lock`) — a read-only mount makes the session die
|
||||
~1 h in. Running as your uid is what makes that write work without a chown.
|
||||
(`make` fills your uid/gid from `id` automatically; for raw `docker compose`, set
|
||||
`NOTEBOOKLM_UID`/`NOTEBOOKLM_GID` in `.env`.)
|
||||
|
||||
## 2. Configure secrets
|
||||
```bash
|
||||
cd deploy && make setup # recommended: picks a tunnel + generates the secrets → .env
|
||||
```
|
||||
Or by hand:
|
||||
```bash
|
||||
cp deploy/.env.example deploy/.env
|
||||
# NOTEBOOKLM_MCP_TOKEN: python -c "import secrets; print(secrets.token_urlsafe(32))"
|
||||
# CF_TUNNEL_TOKEN: from the Cloudflare dashboard (next step)
|
||||
```
|
||||
|
||||
## 3. Choose a tunnel
|
||||
The stack ships two tunnel sidecars as Compose **profiles** — pick one (the server
|
||||
runs under either). Both terminate TLS at their edge, so there's no cert to manage and
|
||||
no host ports are published.
|
||||
|
||||
> **Only using Claude Code / Cursor / Desktop (not claude.ai)? You may not need a public
|
||||
> tunnel at all** — those clients send a bearer, so they need *reachability*, not a public
|
||||
> URL. If the container's host is already on your **Tailscale tailnet** (tailscaled on the
|
||||
> host, not in a sidecar), you can skip Funnel, OAuth, and the MagicDNS-cert / `funnel`
|
||||
> node-attribute steps entirely, and reach the server over the private tailnet:
|
||||
> ```bash
|
||||
> # publish 9420 on the host's TAILSCALE IP only (never 0.0.0.0 — that would expose it
|
||||
> # to your LAN/public), with no tunnel profile:
|
||||
> NOTEBOOKLM_MCP_ALLOW_EXTERNAL_BIND=1 \
|
||||
> docker compose run -d --service-ports -p "$(tailscale ip -4):9420:9420" notebooklm-mcp
|
||||
> claude mcp add --transport http notebooklm \
|
||||
> http://<host>.<your-tailnet>.ts.net:9420/mcp \
|
||||
> --header "Authorization: Bearer $NOTEBOOKLM_MCP_TOKEN"
|
||||
> ```
|
||||
> Most private option — nothing is ever public (plain http is fine; the tailnet is
|
||||
> encrypted). It's a bit more manual than `make up` because it's outside the tunnel
|
||||
> profiles. The tunnels below exist only because **claude.ai** is a cloud client that
|
||||
> needs a public HTTPS URL.
|
||||
|
||||
### 3A. Cloudflare Tunnel (needs a domain in your Cloudflare account)
|
||||
In the Cloudflare dashboard → **Networking → Tunnels → Create Tunnel** (moved there
|
||||
Feb 2026; the old Zero Trust dashboard now redirects here):
|
||||
1. **Create Tunnel** → name it. This is a **Cloudflared** tunnel — hostname ingress; the
|
||||
tunnel's **Type** shows `cloudflared`. Copy the **token** it shows into `CF_TUNNEL_TOKEN`
|
||||
in `.env`. Ignore the `cloudflared` install/run commands on that page — the compose
|
||||
`cloudflared` sidecar runs it for you using the token.
|
||||
2. On the tunnel's **Routes** tab → **Add route** → **Published application**. Under
|
||||
**Hostname**, set a **Subdomain** (e.g. `notebooklm`) + your **Domain** zone → giving
|
||||
`notebooklm.yourdomain.com`; set **Service URL** to `http://notebooklm-mcp:9420` — plain
|
||||
`http` (the compose service listens on HTTP; Cloudflare's edge terminates TLS), not the
|
||||
`https://` in the field's placeholder. Cloudflare auto-creates the DNS record + serves TLS.
|
||||
Leave **Path** empty so the **whole host** (`/`) is routed — not a `/mcp`-scoped path — so
|
||||
the root OAuth routes are reachable. (Profile: `cloudflare`, the default.)
|
||||
> Optional: set `NOTEBOOKLM_MCP_TRUST_PROXY=1` in `.env` to key the per-IP login
|
||||
> throttle on the tunnel's `CF-Connecting-IP` header. Default off keys on the socket
|
||||
> peer (the tunnel egress) — one global throttle bucket. Only enable it behind a trusted
|
||||
> proxy that sets the header; an exposed-directly origin could forge it to dodge the throttle.
|
||||
|
||||
### 3B. Tailscale Funnel (NO domain — free, stable `*.ts.net` HTTPS)
|
||||
Best when you don't own a domain: Tailscale Funnel gives a stable public HTTPS
|
||||
hostname on Tailscale's domain, free on the personal plan, no DNS to manage.
|
||||
**One-time tailnet setup** (admin console — these are policy/feature prerequisites, not
|
||||
per-machine toggles):
|
||||
1. Enable **MagicDNS** and **HTTPS certificates** for the tailnet
|
||||
(admin console → DNS; → HTTPS Certificates).
|
||||
2. Grant the **`funnel` node attribute** in the tailnet policy: admin console →
|
||||
**Settings → Funnel → Manage** (this deep-links to the ACL policy editor) and add a
|
||||
`nodeAttrs` block to the policy file, then **Save** — for a single self-hosted node
|
||||
you can scope it to all members:
|
||||
```json
|
||||
{ "nodeAttrs": [{ "target": ["*"], "attr": ["funnel"] }] }
|
||||
```
|
||||
3. Create a **normal auth key** (Settings → Keys) and put it in `.env` as `TS_AUTHKEY`.
|
||||
(There is no "Funnel-capable" key type — Funnel comes from the policy in step 2.)
|
||||
|
||||
Then the compose `tailscale` sidecar (profile `tailscale`) runs `tailscale/tailscale`
|
||||
with `deploy/tailscale/funnel.json` (`TS_SERVE_CONFIG`), which funnels public `:443 /`
|
||||
→ `notebooklm-mcp:9420` (so the OAuth routes at `/` AND `/mcp` are reachable). The node
|
||||
is `TS_HOSTNAME=notebooklm-mcp`, so your public origin is
|
||||
`https://notebooklm-mcp.<your-tailnet>.ts.net`.
|
||||
> **Find `<your-tailnet>`** on the admin console **DNS** page — the **"Tailnet name"**
|
||||
> shown there (e.g. `tailXXXXXX.ts.net`). After the sidecar is up you can also confirm
|
||||
> the full URL with `docker compose --profile tailscale exec tailscale tailscale serve status`.
|
||||
> Funnel only serves on ports 443/8443/10000 — the config uses **443**, so the URL has
|
||||
> no port suffix. The serve config is bind-mounted as a **directory** (`./tailscale` →
|
||||
> `/config`) per Tailscale's Docker requirement. Not live-verified in this repo —
|
||||
> check `docker compose --profile tailscale logs tailscale` for the served URL on first run.
|
||||
|
||||
Then set the matching **OAuth base URL** in `.env` (bare origin — see step 6):
|
||||
```
|
||||
# Cloudflare: NOTEBOOKLM_MCP_OAUTH_BASE_URL=https://notebooklm.yourdomain.com
|
||||
# Tailscale: NOTEBOOKLM_MCP_OAUTH_BASE_URL=https://notebooklm-mcp.<your-tailnet>.ts.net
|
||||
```
|
||||
|
||||
## 4. Run
|
||||
|
||||
The `Makefile` wraps the tunnel choice (from `make setup`, or `TUNNEL=…`) and the
|
||||
pull-vs-build modes — one command each:
|
||||
|
||||
```bash
|
||||
cd deploy
|
||||
make up # PULL the published image + start (the easy path)
|
||||
make prod VERSION=<version> # ...pin a specific published version
|
||||
make dev # BUILD this checkout + start (contributors)
|
||||
make dev TUNNEL=tailscale # ...forcing the Tailscale Funnel sidecar for this run
|
||||
make logs # tail the server log (expect: bound 0.0.0.0:9420)
|
||||
make restart # rebuild this checkout + recreate after a source change
|
||||
make down # stop and remove
|
||||
```
|
||||
|
||||
`make up` uses this checkout's `pyproject.toml` version. If you copied only
|
||||
`deploy/`, pass `VERSION=<version>` or set `NOTEBOOKLM_MCP_VERSION` in `.env`.
|
||||
If that version has not been published to Docker Hub yet, pass a published
|
||||
`VERSION=<version>` or use `make dev` to build this checkout.
|
||||
|
||||
Equivalent raw compose (`--profile` selects the tunnel; set `NOTEBOOKLM_MCP_VERSION`
|
||||
first, or put it in `.env`):
|
||||
- **Pull + run (any tunnel):** `export NOTEBOOKLM_MCP_VERSION=<version>` then
|
||||
`docker compose --profile cloudflare pull && docker compose --profile cloudflare up -d`
|
||||
(swap `tailscale` for the other tunnel).
|
||||
- **Build from source:** add the build override —
|
||||
`docker compose -f docker-compose.yml -f docker-compose.build.yml --profile cloudflare up -d --build`.
|
||||
- **A different image / tag:** set `NOTEBOOKLM_MCP_IMAGE` / `NOTEBOOKLM_MCP_VERSION` in `.env`.
|
||||
|
||||
## 5. Connect from Claude Code
|
||||
```bash
|
||||
claude mcp add --transport http notebooklm \
|
||||
https://notebooklm-mcp.yourdomain.com/mcp \
|
||||
--header "Authorization: Bearer $NOTEBOOKLM_MCP_TOKEN"
|
||||
```
|
||||
Claude **Desktop** also accepts the bearer. Claude **.ai** (web/mobile) and **ChatGPT**
|
||||
do not — their connector UIs are OAuth-only — so use step 6 (+ step 7 for ChatGPT).
|
||||
|
||||
## 6. (Optional) Connect from claude.ai — self-hosted OAuth (one password)
|
||||
claude.ai's connector UI has no bearer field; it speaks OAuth. Instead of an external
|
||||
IdP, the server runs its own tiny OAuth authorization server gated by **one password**.
|
||||
**Opt-in and additive** — leave both vars unset to stay bearer-only (Claude Code/Desktop
|
||||
unaffected); when set, the bearer and OAuth work side by side on the same `/mcp`.
|
||||
|
||||
1. **`.env`** — set a strong password + your public URL (see `.env.example`):
|
||||
```
|
||||
# a long random secret — the gate (>=16 chars):
|
||||
NOTEBOOKLM_MCP_OAUTH_PASSWORD=$(python -c "import secrets;print(secrets.token_urlsafe(24))")
|
||||
# the BARE public https origin — NOT the /mcp connector URL (the OAuth endpoints
|
||||
# /authorize, /token, /register, /login, /.well-known/* mount at the ROOT):
|
||||
NOTEBOOKLM_MCP_OAUTH_BASE_URL=https://notebooklm.example.com
|
||||
```
|
||||
`make dev` (or `make prod VERSION=…`). Both required together — partial/weak/
|
||||
non-https/has-a-path config refuses to start.
|
||||
2. **Cloudflare tunnel** — the Public Hostname must route the WHOLE host (path `/`,
|
||||
not a `/mcp`-scoped ingress) to `http://notebooklm-mcp:9420`, so the root OAuth
|
||||
routes are reachable. (The `notebooklm.` subdomain is created automatically when you
|
||||
add the Public Hostname; the zone just has to be in your Cloudflare account.)
|
||||
3. **Verify** (after `make dev` + the tunnel is up):
|
||||
```
|
||||
curl https://notebooklm.example.com/.well-known/oauth-authorization-server
|
||||
```
|
||||
`issuer` should be your bare origin and `authorization_endpoint` should be
|
||||
`…/authorize` (at the root). If they show `…/mcp/authorize`, your BASE_URL has the
|
||||
`/mcp` path — drop it.
|
||||
4. **claude.ai → Customize → Connectors** (individual Pro/Max) or **Organization settings →
|
||||
Connectors** (Team/Enterprise owners) → **`+` → Add custom connector** → enter the URL
|
||||
**WITH** `/mcp`: `https://notebooklm.example.com/mcp`. Leave **Advanced settings** (OAuth
|
||||
Client ID/Secret) **blank** — the server supports DCR, so claude.ai registers itself, then
|
||||
opens the server's **password page** in your browser; enter the password → you're connected.
|
||||
Claude Code keeps using the bearer. (Custom connectors work on Free/Pro/Max/Team/Enterprise;
|
||||
Free is capped at one.)
|
||||
|
||||
> **base URL vs connector URL:** `NOTEBOOKLM_MCP_OAUTH_BASE_URL` is the bare origin
|
||||
> (`https://host`); the claude.ai connector URL is that **+ `/mcp`**.
|
||||
|
||||
> **What it does NOT need vs an IdP:** no dashboard, no JWT template, no audience/email
|
||||
> config — the password is the whole identity. Registered clients + tokens **persist**
|
||||
> across restarts in `oauth_state.json` under the mounted profile, so a redeploy doesn't
|
||||
> force re-login. **Treat `oauth_state.json` as a full-account secret** (it holds
|
||||
> long-lived OAuth tokens — same tier as `master_token.json`).
|
||||
> **Honest trade:** because the login page is served through your tunnel, **Cloudflare's
|
||||
> edge sees the password in transit** (it terminates TLS) — use a throwaway Google
|
||||
> account. Note: rotating the password does **not** revoke already-issued OAuth tokens
|
||||
> (they're long-lived + persisted); **real revocation = delete `oauth_state.json` and
|
||||
> restart**. (To remove Cloudflare from the path, self-host TLS instead.)
|
||||
|
||||
## 7. (Optional) Connect from ChatGPT — same OAuth, Developer Mode required
|
||||
ChatGPT's custom MCP connectors speak the **same self-hosted OAuth** as claude.ai — there's
|
||||
no bearer/API-key field, so step 6 (`NOTEBOOKLM_MCP_OAUTH_PASSWORD` + `NOTEBOOKLM_MCP_OAUTH_BASE_URL`)
|
||||
is a prerequisite. It's also **web only** — ChatGPT has no MCP-connector UI on mobile. Which
|
||||
plans get *full* write-capable Developer Mode vs. read/fetch-only MCP changes over time (it has
|
||||
skewed toward Business/Enterprise/Edu for the write tools), so check
|
||||
[OpenAI's Developer Mode doc](https://help.openai.com/en/articles/12584461) for current plan
|
||||
support before relying on the write tools. Then:
|
||||
|
||||
1. **Enable Developer Mode:** ChatGPT → **Settings → Apps & Connectors → Advanced settings →
|
||||
Developer mode** (web only). This unlocks *full* MCP connectors with write tools; without it
|
||||
the connector UI is deep-research (search/fetch) only.
|
||||
2. **Settings → Apps & Connectors** → click **Create** → **MCP Server URL** = the URL **WITH** `/mcp`
|
||||
(`https://notebooklm.example.com/mcp`), **Authentication = OAuth**. ChatGPT registers itself
|
||||
via **DCR** — same as claude.ai, so there's no redirect URI to allowlist on your side — then
|
||||
opens the server's **password page**; enter the password → connected.
|
||||
3. **Don't rely on write prompts.** ChatGPT *may* ask before a write tool runs — it reads each
|
||||
tool's `readOnlyHint`/`destructiveHint` — but whether it prompts depends on the action and your
|
||||
per-connector permission settings, so it isn't a guarantee. The real backstop is server-side:
|
||||
our destructive/sharing-widening tools self-gate on `confirm=true`. Only "always allow" tools on
|
||||
a server you trust — Developer Mode is powerful,
|
||||
and a malicious server or prompt injection can destroy data.
|
||||
|
||||
> Bearer-only deploys (no OAuth) can't be added to ChatGPT **or** claude.ai — both connector UIs
|
||||
> are OAuth-only. Enable step 6 for either.
|
||||
|
||||
## Notes & security
|
||||
- **Two auth layers.** The `NOTEBOOKLM_MCP_TOKEN` bearer gates *who can use the
|
||||
endpoint*; the master token authenticates *the server to Google*. The master
|
||||
token **never** traverses the tunnel — only MCP tool calls/results do. The
|
||||
bearer **does** terminate at Cloudflare (Cloudflare can see it in transit, like
|
||||
any reverse-proxied request), so rotate it freely.
|
||||
- **Fail-closed.** The server refuses to start on a non-loopback bind with no auth
|
||||
at all (neither `NOTEBOOKLM_MCP_TOKEN` nor self-hosted OAuth), and refuses
|
||||
partial/weak/non-https OAuth config.
|
||||
- **One container per account.** Do not scale replicas off one master token —
|
||||
concurrent re-mints invalidate each other's session.
|
||||
- **Rotate the bearer**: change `NOTEBOOKLM_MCP_TOKEN` in `.env`,
|
||||
`docker compose up -d`, and update the `claude mcp add` header.
|
||||
- **Files**: the connector moves text/references only. Add device files via
|
||||
Google Drive (`source_add` with a Drive id) or the NotebookLM app; consume
|
||||
generated podcasts/videos/slides in the NotebookLM app (same account).
|
||||
- **Optional hardening**: instead of a single `rw` bind-mount, mount
|
||||
`master_token.json` as a separate read-only Docker secret and use a writable
|
||||
named volume for `storage_state.json` + `.storage_state.json.lock`.
|
||||
@@ -0,0 +1,14 @@
|
||||
# Override: build the notebooklm-mcp image from THIS checkout instead of pulling
|
||||
# the published one. Layered on top of docker-compose.yml by `make dev`. Raw:
|
||||
# docker compose -f docker-compose.yml -f docker-compose.build.yml \
|
||||
# --profile cloudflare up -d --build
|
||||
#
|
||||
# The build context is the repo root so the image installs this source tree.
|
||||
services:
|
||||
notebooklm-mcp:
|
||||
build:
|
||||
context: ..
|
||||
dockerfile: deploy/Dockerfile
|
||||
# To build from a published PyPI release instead of this checkout:
|
||||
# args:
|
||||
# NOTEBOOKLM_SPEC: "notebooklm-py[mcp,headless]==<version>"
|
||||
@@ -0,0 +1,113 @@
|
||||
# Remote notebooklm-mcp behind a tunnel. Pick ONE tunnel via a Compose profile:
|
||||
#
|
||||
# cp .env.example .env # set NOTEBOOKLM_MCP_TOKEN, NOTEBOOKLM_MCP_VERSION, and/or OAuth
|
||||
# docker compose --profile cloudflare up -d # Cloudflare Tunnel (needs a domain)
|
||||
# docker compose --profile tailscale up -d # Tailscale Funnel (no domain; *.ts.net)
|
||||
# # (the Makefile wraps this: `make dev` / `make dev TUNNEL=tailscale`)
|
||||
#
|
||||
# The notebooklm-mcp service runs under either profile; exactly one tunnel sidecar
|
||||
# is selected. No host ports are published — the ONLY way in is through the chosen
|
||||
# tunnel, which terminates TLS at its edge (no CA cert for you to manage). See README.md.
|
||||
services:
|
||||
notebooklm-mcp:
|
||||
# Pull the published image by default. `make up` / `make prod` require or
|
||||
# inject a version; raw docker compose pull must set NOTEBOOKLM_MCP_VERSION.
|
||||
# Override the namespace/tag via
|
||||
# NOTEBOOKLM_MCP_IMAGE / NOTEBOOKLM_MCP_VERSION in .env. Contributors
|
||||
# building from THIS checkout run `make dev`, which layers
|
||||
# docker-compose.build.yml to add a `build:` section back (see that file).
|
||||
image: ${NOTEBOOKLM_MCP_IMAGE:-tenglin/notebooklm-mcp}:${NOTEBOOKLM_MCP_VERSION:-set-version}
|
||||
restart: unless-stopped
|
||||
# Run as YOUR uid:gid so the mounted profile's files (owned by you on the
|
||||
# host) are readable AND writable with NO chown — and your own `notebooklm`
|
||||
# CLI keeps owning them. Defaults to 1000:1000; `make` fills these from
|
||||
# `id -u`/`id -g` automatically, or set them in .env.
|
||||
user: "${NOTEBOOKLM_UID:-1000}:${NOTEBOOKLM_GID:-1000}"
|
||||
environment:
|
||||
# Bearer token clients present (Authorization: Bearer ...). Env-only. OPTIONAL
|
||||
# here so an OAuth-ONLY deploy (claude.ai, no Claude Code) works — set the bearer
|
||||
# and/or the OAuth vars below. The server's own fail-closed guard refuses to start
|
||||
# on this 0.0.0.0 bind if NEITHER is configured (clear message in the logs).
|
||||
NOTEBOOKLM_MCP_TOKEN: ${NOTEBOOKLM_MCP_TOKEN:-}
|
||||
# Binds 0.0.0.0 (for the cloudflared sidecar) → requires bearer and/or OAuth.
|
||||
NOTEBOOKLM_MCP_ALLOW_EXTERNAL_BIND: "1"
|
||||
# Optional self-hosted OAuth (for claude.ai). Unset → bearer-only; set both
|
||||
# together (partial → refuses to start). See .env.example. The OAuth state
|
||||
# (registered clients + tokens) persists under the mounted profile volume.
|
||||
NOTEBOOKLM_MCP_OAUTH_PASSWORD: ${NOTEBOOKLM_MCP_OAUTH_PASSWORD:-}
|
||||
NOTEBOOKLM_MCP_OAUTH_BASE_URL: ${NOTEBOOKLM_MCP_OAUTH_BASE_URL:-}
|
||||
# Optional (Cloudflare profile only): trust the tunnel's CF-Connecting-IP header as
|
||||
# the per-IP login-throttle key. Default off → throttle keys on the socket peer (the
|
||||
# tunnel egress), i.e. one global bucket. Tailscale Funnel does not set this header.
|
||||
NOTEBOOKLM_MCP_TRUST_PROXY: ${NOTEBOOKLM_MCP_TRUST_PROXY:-}
|
||||
# Point the app at the mounted profile, independent of the (arbitrary) uid's
|
||||
# home. The host dir is always mounted as the profile literally named
|
||||
# "server"; NOTEBOOKLM_HOME/profiles/server is where the app reads+writes.
|
||||
NOTEBOOKLM_HOME: /data
|
||||
NOTEBOOKLM_PROFILE: server
|
||||
# HOME points at the writable mount too, so an arbitrary uid with no
|
||||
# /etc/passwd entry never trips a `~` / getpwuid lookup.
|
||||
HOME: /data/profiles/server
|
||||
volumes:
|
||||
# READ-WRITE: keepalive/recovery rotates cookies into storage_state.json
|
||||
# (+ its .lock). A read-only mount kills the session ~1h in.
|
||||
# Defaults to your local "default" profile; override with NOTEBOOKLM_PROFILE_DIR
|
||||
# in .env to point at a different profile dir (e.g. a dedicated/throwaway one).
|
||||
# Treat the dir as a secret — master_token.json is a full-account credential.
|
||||
- "${NOTEBOOKLM_PROFILE_DIR:-${HOME}/.notebooklm/profiles/default}:/data/profiles/server:rw"
|
||||
# No `ports:` — not reachable except via the tunnel.
|
||||
|
||||
# --- Tunnel option A: Cloudflare (needs a domain in your Cloudflare account) ----
|
||||
cloudflared:
|
||||
profiles: ["cloudflare"]
|
||||
# Pinnable for reproducibility: set CLOUDFLARED_VERSION in .env to a release
|
||||
# from github.com/cloudflare/cloudflared/releases. Defaults to `latest` (the
|
||||
# tunnel connector is backward-compatible and is what Cloudflare recommends
|
||||
# running), so the template works without choosing a tag.
|
||||
image: cloudflare/cloudflared:${CLOUDFLARED_VERSION:-latest}
|
||||
restart: unless-stopped
|
||||
command: tunnel run
|
||||
environment:
|
||||
# From the Cloudflare Zero Trust dashboard (Networks → Tunnels). Configure
|
||||
# the tunnel's Public Hostname to point at http://notebooklm-mcp:9420.
|
||||
# `:-` not `:?` — Compose interpolates the WHOLE file before profile filtering,
|
||||
# so a hard-required `:?` here would error even under `--profile tailscale`.
|
||||
# cloudflared fails loudly in its own logs if this is empty when the profile runs.
|
||||
TUNNEL_TOKEN: ${CF_TUNNEL_TOKEN:-}
|
||||
depends_on:
|
||||
# Wait until the MCP server passes its HEALTHCHECK (port listening), not
|
||||
# merely until its container starts, before bringing the tunnel up.
|
||||
notebooklm-mcp:
|
||||
condition: service_healthy
|
||||
|
||||
# --- Tunnel option B: Tailscale Funnel (NO domain; free stable *.ts.net HTTPS) --
|
||||
# tailscale/tailscale Funnel-in-Docker pattern (see Tailscale's Docker docs). It
|
||||
# publishes the WHOLE host root → notebooklm-mcp:9420 (so the OAuth routes at / AND
|
||||
# /mcp are reachable). Your BASE_URL becomes https://<TS_HOSTNAME>.<tailnet>.ts.net.
|
||||
# PREREQS (one-time, in the Tailscale admin console): enable MagicDNS + HTTPS certs,
|
||||
# and grant the `funnel` node attribute in the tailnet policy (Access controls →
|
||||
# Funnel → "Add Funnel to policy"). TS_AUTHKEY is a normal auth key — Funnel comes
|
||||
# from the policy, not the key. NOTE: not live-verified in this repo; check
|
||||
# `docker compose --profile tailscale logs tailscale` for the served URL.
|
||||
tailscale:
|
||||
profiles: ["tailscale"]
|
||||
image: tailscale/tailscale:latest
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
# `:-` not `:?` (see the cloudflared note) — required only when the tailscale
|
||||
# profile actually runs; the container errors clearly if it's empty then.
|
||||
TS_AUTHKEY: ${TS_AUTHKEY:-}
|
||||
TS_HOSTNAME: notebooklm-mcp # → https://notebooklm-mcp.<your-tailnet>.ts.net
|
||||
TS_STATE_DIR: /var/lib/tailscale
|
||||
TS_USERSPACE: "true" # no NET_ADMIN / /dev/net/tun needed
|
||||
TS_SERVE_CONFIG: /config/funnel.json
|
||||
volumes:
|
||||
- tailscale-state:/var/lib/tailscale
|
||||
# MUST mount the serve-config DIR (not the file) so tailscaled detects/reads it.
|
||||
- ./tailscale:/config:ro
|
||||
depends_on:
|
||||
notebooklm-mcp:
|
||||
condition: service_healthy
|
||||
|
||||
volumes:
|
||||
tailscale-state:
|
||||
Executable
+82
@@ -0,0 +1,82 @@
|
||||
#!/usr/bin/env sh
|
||||
# Interactive first-run setup for the remote notebooklm-mcp deploy.
|
||||
# Picks a tunnel, generates strong secrets, and writes deploy/.env.
|
||||
# Invoked by `make setup`. Re-runnable only after you remove .env.
|
||||
set -eu
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
|
||||
if [ -e .env ]; then
|
||||
echo "✗ deploy/.env already exists — refusing to overwrite it (it may hold secrets)."
|
||||
echo " Edit it by hand, or 'rm .env' first to start over."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Find a Python 3 with the stdlib `secrets` module (>=3.6). Probing the import
|
||||
# (not just `command -v`) rejects a `python` that is really Python 2.
|
||||
PY=""
|
||||
for c in python3 python; do
|
||||
if "$c" -c "import secrets" >/dev/null 2>&1; then PY="$c"; break; fi
|
||||
done
|
||||
[ -n "$PY" ] || { printf '%s\n' "✗ Need python3 (with the secrets module) on PATH."; exit 1; }
|
||||
gen() { "$PY" -c "import secrets; print(secrets.token_urlsafe($1))"; }
|
||||
|
||||
printf 'Tunnel — [1] Cloudflare (needs a domain) [2] Tailscale Funnel (no domain) [1]: '
|
||||
read -r choice
|
||||
case "$choice" in
|
||||
2 | tailscale | ts) TUNNEL=tailscale ;;
|
||||
*) TUNNEL=cloudflare ;;
|
||||
esac
|
||||
|
||||
printf 'Also connect from claude.ai (self-hosted OAuth, one password)? [y/N]: '
|
||||
read -r want_oauth
|
||||
case "$want_oauth" in
|
||||
y | Y | yes) OAUTH=1 ;;
|
||||
*) OAUTH= ;;
|
||||
esac
|
||||
|
||||
if [ "$TUNNEL" = cloudflare ]; then
|
||||
printf 'CF_TUNNEL_TOKEN (Cloudflare Zero Trust → Networks → Tunnels; blank to fill later): '
|
||||
else
|
||||
printf 'TS_AUTHKEY (Tailscale → Settings → Keys; blank to fill later): '
|
||||
fi
|
||||
read -r TUN_TOKEN
|
||||
|
||||
BASE_URL=
|
||||
if [ -n "$OAUTH" ]; then
|
||||
printf 'Public https origin claude.ai reaches (e.g. https://host.example.com): '
|
||||
read -r BASE_URL
|
||||
BASE_URL=${BASE_URL%/} # normalize to a bare origin (server rejects a trailing path)
|
||||
fi
|
||||
|
||||
TOKEN=$(gen 32)
|
||||
OAUTH_PW=$(gen 24)
|
||||
|
||||
# .env holds full-account secrets — create it 0600 (umask before the redirect, so
|
||||
# there's no world-readable window between create and a later chmod).
|
||||
umask 077
|
||||
{
|
||||
# printf '%s', never echo — a token can begin with '-' or (elsewhere) contain
|
||||
# backslashes, both of which some echo implementations would mangle.
|
||||
printf '%s\n' "# Generated by 'make setup'. Treat as secret (gitignored). See .env.example for all options."
|
||||
printf 'TUNNEL=%s\n' "$TUNNEL"
|
||||
printf 'NOTEBOOKLM_MCP_TOKEN=%s\n' "$TOKEN"
|
||||
if [ -n "$OAUTH" ]; then
|
||||
printf 'NOTEBOOKLM_MCP_OAUTH_PASSWORD=%s\n' "$OAUTH_PW"
|
||||
printf 'NOTEBOOKLM_MCP_OAUTH_BASE_URL=%s\n' "$BASE_URL"
|
||||
fi
|
||||
if [ "$TUNNEL" = cloudflare ]; then
|
||||
printf 'CF_TUNNEL_TOKEN=%s\n' "$TUN_TOKEN"
|
||||
[ -n "$OAUTH" ] && printf '%s\n' "NOTEBOOKLM_MCP_TRUST_PROXY=1"
|
||||
else
|
||||
printf 'TS_AUTHKEY=%s\n' "$TUN_TOKEN"
|
||||
fi
|
||||
} > .env
|
||||
|
||||
echo
|
||||
echo "✓ Wrote deploy/.env — TUNNEL=$TUNNEL, bearer token generated${OAUTH:+, OAuth password generated}."
|
||||
echo "Next:"
|
||||
echo " 1. Bootstrap the master token once (deploy/README.md §1) if you haven't."
|
||||
echo " 2. Finish the tunnel setup (deploy/README.md §3)."
|
||||
[ -z "$TUN_TOKEN" ] && echo " …then put the tunnel token in deploy/.env."
|
||||
echo " 3. make up"
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"TCP": {
|
||||
"443": { "HTTPS": true }
|
||||
},
|
||||
"Web": {
|
||||
"${TS_CERT_DOMAIN}:443": {
|
||||
"Handlers": {
|
||||
"/": { "Proxy": "http://notebooklm-mcp:9420" }
|
||||
}
|
||||
}
|
||||
},
|
||||
"AllowFunnel": {
|
||||
"${TS_CERT_DOMAIN}:443": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
# Keep the packed .mcpb hermetic: never ship compiled-bytecode cruft even if a
|
||||
# build ran run_server.py first (e.g. the unit tests import it by path, which
|
||||
# writes __pycache__/). CI packs from a fresh checkout, but this makes the
|
||||
# artifact deterministic regardless of build-env state.
|
||||
__pycache__/
|
||||
*.pyc
|
||||
@@ -0,0 +1,89 @@
|
||||
# NotebookLM MCP — Claude Desktop extension (`.mcpb` / DXT)
|
||||
|
||||
This directory packages the [`notebooklm-py`](https://github.com/teng-lin/notebooklm-py)
|
||||
MCP server as a one-click [Claude Desktop extension](https://www.anthropic.com/engineering/desktop-extensions)
|
||||
(`.mcpb`, formerly DXT). Installing the bundle wires the NotebookLM tools into
|
||||
Claude Desktop without any manual JSON editing.
|
||||
|
||||
## Contents
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `manifest.json` | The extension manifest (name, version, server command). |
|
||||
| `run_server.py` | A resilient launcher that locates `uvx` and runs `uvx --from "notebooklm-py[mcp]" notebooklm-mcp`, forwarding stdin/stdout/stderr for clean JSON-RPC. |
|
||||
|
||||
The launcher resolves the server from PyPI on demand via `uvx`, so there is no
|
||||
vendored Python environment in the bundle — only the two files above.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. **`uv` / `uvx`** — the launcher shells out to `uvx`:
|
||||
```bash
|
||||
# macOS / Linux
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
# Windows (PowerShell)
|
||||
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
|
||||
```
|
||||
`run_server.py` searches `PATH` plus the common install dirs
|
||||
(`~/.local/bin`, `~/.cargo/bin`, `/opt/homebrew/bin`, `/usr/local/bin`,
|
||||
`/snap/bin`, and the Windows equivalents), so a restricted host `PATH` is fine.
|
||||
|
||||
2. **Authenticate once** with your Google account:
|
||||
```bash
|
||||
uvx --from "notebooklm-py[mcp]" notebooklm login
|
||||
```
|
||||
This stores credentials under `~/.notebooklm/` for the active profile, which
|
||||
the server binds at startup.
|
||||
|
||||
## Install
|
||||
|
||||
- **Claude Desktop:** download `notebooklm-mcp.mcpb` from the
|
||||
[latest release](https://github.com/teng-lin/notebooklm-py/releases/latest)
|
||||
(under **Assets**), then double-click it — or *Settings → Extensions →
|
||||
Install Extension…* and pick the file. Restart Claude Desktop; the NotebookLM
|
||||
tools (e.g. `notebook_list`, `chat_ask`, `studio_generate`) appear in the tool
|
||||
picker.
|
||||
|
||||
Each stable release attaches a prebuilt, version-matched bundle (built and
|
||||
uploaded by `.github/workflows/publish-mcpb.yml`), so there is nothing to
|
||||
build yourself. (Pre-releases don't ship a bundle — the launcher resolves the
|
||||
latest stable server from PyPI regardless.)
|
||||
|
||||
For other MCP clients (Claude Code, Cursor, Windsurf) that read a JSON config
|
||||
instead of a `.mcpb`, use the CLI installer instead:
|
||||
|
||||
```bash
|
||||
notebooklm mcp install claude-code # or: cursor / windsurf / claude-desktop
|
||||
```
|
||||
|
||||
## Build from source (contributors)
|
||||
|
||||
End users should download the prebuilt bundle from the release above. If you are
|
||||
developing the extension itself, build it locally with the official
|
||||
[`@anthropic-ai/mcpb`](https://github.com/anthropics/mcpb) CLI (formerly `dxt`):
|
||||
|
||||
```bash
|
||||
# From the repository root:
|
||||
npx @anthropic-ai/mcpb pack desktop-extension notebooklm-mcp.mcpb
|
||||
```
|
||||
|
||||
This produces `notebooklm-mcp.mcpb`, a zip of `manifest.json` + `run_server.py` —
|
||||
the same artifact the release workflow attaches.
|
||||
|
||||
> The CLI validates `manifest.json` against the DXT schema during `pack`. The
|
||||
> repo's own `tests/unit/test_mcp_desktop_extension.py` asserts the manifest is
|
||||
> valid JSON with the required keys, that its version tracks the package version,
|
||||
> and that `run_server.py` builds the correct `uvx` command, so the bundle stays
|
||||
> shippable.
|
||||
|
||||
## How the launcher passes through stdio
|
||||
|
||||
The MCP host talks to the server over **JSON-RPC on stdin/stdout**. `run_server.py`
|
||||
therefore:
|
||||
|
||||
- prints **nothing** to stdout (diagnostics go to stderr only), and
|
||||
- passes the host's `stdin`/`stdout`/`stderr` straight through to the child
|
||||
`uvx` process.
|
||||
|
||||
If `uvx` cannot be found, the launcher prints a clear install hint to **stderr**
|
||||
and exits non-zero — it never contaminates the JSON-RPC channel.
|
||||
@@ -0,0 +1,84 @@
|
||||
{
|
||||
"manifest_version": "0.3",
|
||||
"name": "notebooklm-mcp",
|
||||
"display_name": "NotebookLM (notebooklm-py)",
|
||||
"version": "0.8.0a3",
|
||||
"description": "Connect Claude to Google NotebookLM: manage notebooks and sources, chat over your notebooks, generate podcasts/videos/quizzes/mind maps, and run research.",
|
||||
"long_description": "This extension connects an MCP host (e.g. Claude Desktop) to Google NotebookLM via the Model Context Protocol, backed by the `notebooklm-py` Python client.\n\n**Prerequisites:**\n1. Install `uv` (the launcher runs `uvx`): `curl -LsSf https://astral.sh/uv/install.sh | sh`\n2. Authenticate once: `uvx --from \"notebooklm-py[mcp]\" notebooklm login` (or `notebooklm login` if installed), then select your Google profile.\n\nThe bundled `run_server.py` launcher locates `uvx` across common install paths and runs `uvx --from \"notebooklm-py[mcp]\" notebooklm-mcp`, so no global install is required.\n\n**Features:**\n- Create, list, describe, rename, and delete notebooks\n- Add sources (URL, YouTube, Google Drive, text, files) and read their content\n- Chat / ask questions over a notebook\n- Generate Audio Overviews (podcasts), videos, quizzes, flashcards, reports, and mind maps\n- Research topics via web or Drive and import the results",
|
||||
"author": {
|
||||
"name": "Teng Lin",
|
||||
"email": "teng.lin@gmail.com",
|
||||
"url": "https://github.com/teng-lin"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "https://github.com/teng-lin/notebooklm-py"
|
||||
},
|
||||
"homepage": "https://github.com/teng-lin/notebooklm-py",
|
||||
"documentation": "https://github.com/teng-lin/notebooklm-py#readme",
|
||||
"support": "https://github.com/teng-lin/notebooklm-py/issues",
|
||||
"server": {
|
||||
"type": "python",
|
||||
"entry_point": "run_server.py",
|
||||
"mcp_config": {
|
||||
"command": "python3",
|
||||
"args": [
|
||||
"${__dirname}/run_server.py"
|
||||
],
|
||||
"env": {},
|
||||
"platform_overrides": {
|
||||
"win32": {
|
||||
"command": "python"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"tools": [
|
||||
{
|
||||
"name": "notebook_list",
|
||||
"description": "List all notebooks"
|
||||
},
|
||||
{
|
||||
"name": "notebook_create",
|
||||
"description": "Create a new notebook"
|
||||
},
|
||||
{
|
||||
"name": "source_add",
|
||||
"description": "Add a source (URL, YouTube, Drive, text, or file)"
|
||||
},
|
||||
{
|
||||
"name": "chat_ask",
|
||||
"description": "Ask a question over a notebook's sources"
|
||||
},
|
||||
{
|
||||
"name": "studio_generate",
|
||||
"description": "Generate audio, video, quizzes, flashcards, reports, or mind maps"
|
||||
},
|
||||
{
|
||||
"name": "research_start",
|
||||
"description": "Research a topic via web or Drive search"
|
||||
}
|
||||
],
|
||||
"tools_generated": true,
|
||||
"keywords": [
|
||||
"notebooklm",
|
||||
"google",
|
||||
"podcast",
|
||||
"research",
|
||||
"notebook",
|
||||
"ai",
|
||||
"mcp"
|
||||
],
|
||||
"license": "MIT",
|
||||
"compatibility": {
|
||||
"claude_desktop": ">=0.10.0",
|
||||
"platforms": [
|
||||
"darwin",
|
||||
"win32",
|
||||
"linux"
|
||||
],
|
||||
"runtimes": {
|
||||
"python": ">=3.10"
|
||||
}
|
||||
}
|
||||
}
|
||||
Executable
+169
@@ -0,0 +1,169 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Resilient launcher for the NotebookLM MCP server in Claude Desktop.
|
||||
|
||||
Claude Desktop (and similar MCP hosts) often run with a *restricted* ``PATH``
|
||||
that omits user-local install dirs, so ``uvx`` may not be discoverable via the
|
||||
shell ``PATH`` alone. This launcher searches the common install locations before
|
||||
falling back to ``PATH``, then execs::
|
||||
|
||||
uvx --from "notebooklm-py[mcp]" notebooklm-mcp [<forwarded args>]
|
||||
|
||||
forwarding the host's stdin/stdout/stderr through cleanly. The stdio passthrough
|
||||
is critical: the MCP host speaks JSON-RPC over the child's stdin/stdout, so this
|
||||
launcher must never write its own bytes to stdout. Diagnostics go to STDERR
|
||||
only, and a missing ``uvx`` exits non-zero with a clear message.
|
||||
|
||||
Bundled inside the ``.mcpb`` extension and invoked via ``manifest.json``::
|
||||
|
||||
"command": "python3", "args": ["${__dirname}/run_server.py"]
|
||||
|
||||
The module exposes ``find_uvx`` / ``build_command`` / ``main`` as importable
|
||||
functions so the bundle can be unit-tested without execing anything.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
#: The PyPI distribution + ``mcp`` extra to resolve, and the console script to
|
||||
#: run. ``uvx --from "<PACKAGE>" <CONSOLE_SCRIPT>`` fetches the package (with the
|
||||
#: extra) into an ephemeral environment and runs the script.
|
||||
PACKAGE = "notebooklm-py[mcp]"
|
||||
CONSOLE_SCRIPT = "notebooklm-mcp"
|
||||
|
||||
|
||||
def _bundle_version() -> str | None:
|
||||
"""Read this bundle's version from the sibling ``manifest.json``.
|
||||
|
||||
Returns the version string, or ``None`` when the manifest is missing or
|
||||
unreadable (a defensive fallback — a bundle always ships ``manifest.json``
|
||||
beside this launcher).
|
||||
"""
|
||||
manifest = os.path.join(os.path.dirname(os.path.abspath(__file__)), "manifest.json")
|
||||
try:
|
||||
with open(manifest, encoding="utf-8") as fh:
|
||||
data = json.load(fh)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return None
|
||||
version = data.get("version") if isinstance(data, dict) else None
|
||||
return version if isinstance(version, str) and version else None
|
||||
|
||||
|
||||
def _is_prerelease(version: str) -> bool:
|
||||
"""True for a PEP 440 pre-release or development version.
|
||||
|
||||
Covers ``…aN`` / ``…bN`` / ``…rcN`` and the alternate spellings
|
||||
(``alpha`` / ``beta`` / ``c`` / ``pre`` / ``preview``), a trailing ``.devN``
|
||||
(including on top of a pre-release, e.g. ``0.8.0a1.dev0``), and implicit-zero
|
||||
forms (``0.8.0a``). A clean or ``.postN`` release is not a pre-release.
|
||||
"""
|
||||
return (
|
||||
re.search(
|
||||
r"[-._]?(?:a|b|rc|alpha|beta|c|pre|preview|dev)[-._]?\d*(?:[-._]?dev[-._]?\d*)?$",
|
||||
version,
|
||||
)
|
||||
is not None
|
||||
)
|
||||
|
||||
|
||||
def _candidate_uvx_paths() -> list[str]:
|
||||
"""Return common ``uvx`` install locations for the current platform."""
|
||||
home = os.path.expanduser("~")
|
||||
system = platform.system()
|
||||
|
||||
if system == "Windows":
|
||||
appdata = os.environ.get("LOCALAPPDATA", os.path.join(home, "AppData", "Local"))
|
||||
return [
|
||||
os.path.join(home, ".local", "bin", "uvx.exe"),
|
||||
os.path.join(home, ".cargo", "bin", "uvx.exe"),
|
||||
os.path.join(appdata, "uv", "uvx.exe"),
|
||||
os.path.join(home, "scoop", "shims", "uvx.exe"),
|
||||
]
|
||||
|
||||
# POSIX (macOS + Linux): the uv installer drops uvx in ~/.local/bin or
|
||||
# ~/.cargo/bin; Homebrew on Apple Silicon uses /opt/homebrew/bin; Intel macOS
|
||||
# and most Linux package managers use /usr/local/bin. /snap/bin covers the
|
||||
# Ubuntu snap.
|
||||
return [
|
||||
os.path.join(home, ".local", "bin", "uvx"),
|
||||
os.path.join(home, ".cargo", "bin", "uvx"),
|
||||
"/opt/homebrew/bin/uvx",
|
||||
"/usr/local/bin/uvx",
|
||||
"/snap/bin/uvx",
|
||||
]
|
||||
|
||||
|
||||
def find_uvx() -> str | None:
|
||||
"""Locate the ``uvx`` executable.
|
||||
|
||||
Checks ``PATH`` first (honors a host that already exposes ``uvx``), then the
|
||||
common per-platform install locations. Returns the absolute path, or
|
||||
``None`` when ``uvx`` cannot be found anywhere.
|
||||
"""
|
||||
on_path = shutil.which("uvx")
|
||||
if on_path:
|
||||
return on_path
|
||||
|
||||
for candidate in _candidate_uvx_paths():
|
||||
if os.path.isfile(candidate) and os.access(candidate, os.X_OK):
|
||||
return candidate
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def build_command(uvx: str, argv: list[str], version: str | None = None) -> list[str]:
|
||||
"""Build the ``uvx`` exec argv, forwarding ``argv`` to the console script.
|
||||
|
||||
A **stable** bundle stays unpinned so it tracks the latest stable server. A
|
||||
**pre-release** bundle pins its exact ``version`` (e.g.
|
||||
``notebooklm-py[mcp]==0.8.0a1``) so ``uvx`` launches that pre-release rather
|
||||
than the latest stable. The explicit ``==`` pre-release pin is enough — uv's
|
||||
default resolver accepts a pre-release requested by an explicit marker — so
|
||||
no ``--prerelease`` flag is used; ``--prerelease=allow`` would needlessly let
|
||||
*transitive* dependencies resolve to pre-releases too, hurting reproducibility.
|
||||
"""
|
||||
spec = f"{PACKAGE}=={version}" if version is not None and _is_prerelease(version) else PACKAGE
|
||||
return [uvx, "--from", spec, CONSOLE_SCRIPT, *argv]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Find ``uvx`` and exec the NotebookLM MCP server, or fail cleanly."""
|
||||
uvx = find_uvx()
|
||||
|
||||
if not uvx:
|
||||
# STDERR only — stdout is the JSON-RPC channel and must stay pristine.
|
||||
print(
|
||||
"Error: could not find 'uvx'. Install uv first:\n"
|
||||
" curl -LsSf https://astral.sh/uv/install.sh | sh\n"
|
||||
' # Windows: powershell -c "irm https://astral.sh/uv/install.ps1 | iex"\n'
|
||||
"\n"
|
||||
"Then restart your MCP host (e.g. Claude Desktop).",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
|
||||
# Explicit stdin/stdout/stderr passthrough is critical: the MCP host
|
||||
# communicates with the server via stdin/stdout JSON-RPC. We do NOT capture
|
||||
# them — they flow straight through to the child process.
|
||||
cmd = build_command(uvx, sys.argv[1:], _bundle_version())
|
||||
try:
|
||||
result = subprocess.run( # noqa: S603 - argv is constructed, not shell-interpolated
|
||||
cmd,
|
||||
stdin=sys.stdin,
|
||||
stdout=sys.stdout,
|
||||
stderr=sys.stderr,
|
||||
check=False,
|
||||
)
|
||||
except KeyboardInterrupt:
|
||||
sys.exit(130)
|
||||
sys.exit(result.returncode)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,34 @@
|
||||
# ADR-NNNN: <Title>
|
||||
|
||||
## Status
|
||||
|
||||
One of: `Proposed`, `Proposed — <short explanation>`, `Accepted`, `Accepted (retroactive)`, `Accepted (#PR)`, `Accepted (Sunset = <event>)`, `Superseded — <short explanation>`, `Superseded by ADR-NNNN (#PR)`, `Deprecated`, `Rejected`.
|
||||
|
||||
A *sunset clause* records the conditions under which the decision is expected to be revisited (e.g. "Accepted (Sunset = D2 cutover)" means the ADR is in force today but will be re-statused when the named work lands).
|
||||
|
||||
## Context
|
||||
|
||||
What is the problem? What forces are in play (technical, organizational, historical)? What constraints does the codebase impose that make this decision non-obvious? Cite specific files / line ranges so a future reader can find the artifact the decision shaped.
|
||||
|
||||
Keep this section short and load-bearing. Anything that does not help the reader make the same call again is noise.
|
||||
|
||||
## Decision
|
||||
|
||||
The choice we made, stated as a single declarative sentence (or a tight bulleted list when the decision has multiple inseparable parts). No hedging — the ADR is the place where the team's intent is unambiguous.
|
||||
|
||||
## Consequences
|
||||
|
||||
What follows from the decision. Include both wanted and unwanted effects: maintenance cost, test patterns, performance, ergonomics, deprecation paths. Be honest about the downsides — an ADR that lists only benefits is a marketing brief, not a decision record.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
Each rejected alternative, one short paragraph, explaining *why it was rejected*. The bar is "would a future reader, lacking this ADR, plausibly re-propose this alternative?" If yes, it belongs here.
|
||||
|
||||
---
|
||||
|
||||
**Authoring notes (delete on copy):**
|
||||
|
||||
- Numbering is append-only. Never re-use a retired number.
|
||||
- Use lowercase-kebab-case filenames: `NNNN-short-title.md`.
|
||||
- Update `docs/adr/README.md` index when adding or re-statusing an ADR.
|
||||
- Keep ADRs short (target: ≤ 250 lines). If a decision needs more, it probably needs to split.
|
||||
@@ -0,0 +1,109 @@
|
||||
# ADR-0001: Layered `_core` seams and the property-bridge policy
|
||||
|
||||
> **Current state (2026-05-29).** This ADR is **Superseded** and documents a
|
||||
> retired pattern for historical context only. The `_core.py` monolith it
|
||||
> describes was decomposed and **deleted long ago**; the later `Session`
|
||||
> facade and `_session.py` that replaced it have **also been deleted**, and the
|
||||
> `_session_*.py` collaborators moved into the `_runtime/` package and
|
||||
> sibling runtime modules. No
|
||||
> `_core.py`/`_session.py` file exists today — read every in-body
|
||||
> `src/notebooklm/_core.py:NNN` reference and `Session` mention as historical.
|
||||
> The live runtime decomposition is documented in
|
||||
> [`docs/architecture.md`](../architecture.md) and
|
||||
> [`CLAUDE.md`](../../CLAUDE.md); the empty compat-bridge allowlist guard is
|
||||
> [`tests/_guardrails/test_no_session_compat_bridges.py`](../../tests/_guardrails/test_no_session_compat_bridges.py).
|
||||
|
||||
## Status
|
||||
|
||||
Superseded — bridge policy retired in the session-shrink arc (PRs 0-7).
|
||||
|
||||
Documents a pattern shipped incrementally across tier-1 through tier-10
|
||||
(PRs roughly mid-2025 through 2026-05). This ADR backfills the rationale
|
||||
for the temporary property-bridge policy. The policy's built-in exit
|
||||
condition ("Be retired the moment its only readers are themselves
|
||||
retired") has now been satisfied: the staged session-shrink arc removed
|
||||
the `Session` property shims and their test readers. The permanent guard
|
||||
is [`tests/_guardrails/test_no_session_compat_bridges.py`](../../tests/_guardrails/test_no_session_compat_bridges.py),
|
||||
whose allowlist is empty.
|
||||
|
||||
## Context
|
||||
|
||||
`NotebookLMClient` is an `async` client for an undocumented Google RPC surface (`batchexecute`). The earliest implementation co-located transport, RPC dispatch, auth refresh, drain coordination, metrics, request-id assignment, cookie persistence, lifecycle, polling, and conversation caching inside one `_core.py` module. By the time the codebase reached tier 7 the module had crossed ~1,800 lines and 90+ methods. Two pressures broke the monolith:
|
||||
|
||||
- **Independent testability** — the auth-refresh loop, the keepalive task, and the drain coordinator each have non-trivial timing semantics. Testing them through `NotebookLMClient` required spinning the full client and patching half a dozen unrelated collaborators per test.
|
||||
- **Independent reasoning** — landlocked invariants (e.g. "drain must complete before close returns") had no module-level home, so reviewers had to re-derive them on every PR.
|
||||
|
||||
Tier 8/9/10 extracted the cross-cutting concerns into named seam modules. At the tier-10 baseline the seams were:
|
||||
|
||||
```text
|
||||
_request_types.py authed-POST request construction types
|
||||
_transport_errors.py transport exceptions + POST error mapping
|
||||
_streaming_post.py size-capped streaming POST helper
|
||||
_rpc_executor.py RPC dispatch executor (DecodeResponse, RpcOwner protocols)
|
||||
_session_auth.py AuthRefreshCoordinator + auth-snapshot lock
|
||||
_transport_drain.py TransportDrainTracker + _TransportOperationToken
|
||||
_client_metrics.py ClientMetrics counters + on_rpc_event callback
|
||||
_reqid_counter.py ReqidCounter (monotonic _reqid for chat backend)
|
||||
_conversation_cache.py Per-instance LRU conversation cache
|
||||
_polling_registry.py Pending-poll registry for long-running artifact gens
|
||||
_session_lifecycle.py Open/close lifecycle (loop-affinity guard + keepalive task)
|
||||
_cookie_persistence.py Cookie-jar persistence + __Secure-1PSIDTS rotation
|
||||
_session_config.py Module-level DEFAULT_* knobs
|
||||
_session_helpers.py is_auth_error / AUTH_ERROR_PATTERNS / keepalive helpers
|
||||
_error_injection.py env-var guard (live implementation is chain-level `ErrorInjectionMiddleware` in `src/notebooklm/_middleware/error_injection.py`; `_SyntheticErrorTransport` retired)
|
||||
```
|
||||
|
||||
The extraction was constrained by an unusually high test-coupling load: tests reach into the live `Session` instance with `core._save_lock`, `core._metrics_lock`, `core._on_rpc_event`, and many other private attributes — patterns that pre-date the seam extraction. When the storage for these attributes moved into the seams, the legacy attribute names had to keep resolving on the `Session` instance or hundreds of tests would break in a single PR.
|
||||
|
||||
The chosen mechanism is a *property-bridge policy*. Each migrated attribute keeps its legacy name on `Session`, but the property delegates reads and writes to the owning seam:
|
||||
|
||||
```python
|
||||
# src/notebooklm/_core.py:450-774 (representative excerpt)
|
||||
@property
|
||||
def _save_lock(self) -> threading.Lock:
|
||||
return self.cookie_persistence.save_lock
|
||||
|
||||
@_save_lock.setter
|
||||
def _save_lock(self, value: threading.Lock) -> None:
|
||||
self.cookie_persistence.save_lock = value
|
||||
```
|
||||
|
||||
Roughly 324 lines of `_core.py` (lines ~450-774) are property bridges of this form. They exist for two distinct reasons:
|
||||
|
||||
1. **Sub-client compatibility** — the capability adapter (`_capabilities.py`) refers to attribute names that have been physically extracted; the bridge keeps the adapter's contract intact.
|
||||
2. **Test compatibility** — `monkeypatch.setattr(core, "_save_lock", fake_lock)` is a load-bearing test idiom across ~273 sites (see ADR-0003 and the forthcoming ADR-0007); the bridge makes such patches write through to the real storage.
|
||||
|
||||
## Decision
|
||||
|
||||
`Session` is decomposed into thirteen seam modules, one per cross-cutting concern. `Session` itself becomes an orchestrator that wires the seams together and owns the public RPC surface.
|
||||
|
||||
Property bridges in `_core.py` are *permitted but tracked*. Each bridge must:
|
||||
|
||||
- Be added only when the attribute being relocated has live external readers (tests, sub-clients, downstream code).
|
||||
- Read and write through to the owning seam — never store state of its own.
|
||||
- Be retired the moment its only readers are themselves retired (see ADR-0002 for the sub-client-compat half; ADR-0007 will cover the test-compat half).
|
||||
|
||||
The seam extractions are behavior-preserving moves. Each one ships with a unit-test fixture that exercises the seam in isolation; nothing on the seam should require a full `Session` to test.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- Each seam can be reviewed, tested, and refactored without touching the others. The drain coordinator, the keepalive loop, and the metrics counters now have file-local invariants.
|
||||
- The orchestrator (`Session`) makes the wiring graph readable in one place. The pattern is "instantiate seams in `__init__`, expose them as plain attributes, delegate public methods through them."
|
||||
- The seam extraction is reversible. Because each seam owns one concept, mistakes can be undone without unrolling unrelated work.
|
||||
|
||||
**Unwanted:**
|
||||
|
||||
- Historical: `_core.py` stayed large even after extraction because of
|
||||
the property-bridge zoo. The bridges were the *exit cost* of the
|
||||
architecture's test patterns, not the seam pattern itself. ADR-0002,
|
||||
ADR-0007, and the later session-shrink arc supplied the removal path.
|
||||
- The seam protocols (`RpcOwner`, `DecodeResponse`, feature runtime Protocols, etc.) introduce extra type surface. This pays for itself only because the seams are tested independently — if the seams collapsed back into `_core.py`, the protocols would be ceremonial.
|
||||
- New contributors must learn the seam map. The CLAUDE.md "Repository Structure" section exists for this reason; it should remain a thin onboarding guide rather than a duplicate of this ADR.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Keep `_core.py` monolithic.** Rejected. The module had crossed the maintainability threshold (90+ methods, mixed concerns, mixed locking strategies, mixed lifetimes). Reviewers were re-deriving the same invariants on every PR; the cost was paid on every change, not just on the rare refactor.
|
||||
- **Extract into a sibling package (`src/notebooklm/core/`).** Rejected at the time because the seams have a clear single owner (`Session`) and a sibling package implies "multiple consumers" — which is not (yet) the situation. The seams may be promoted to a sub-package in a future tier if a second consumer appears; the current flat layout keeps the cognitive load low.
|
||||
- **Skip property bridges, accept the test breakage.** Rejected. An internal architecture audit measured ~273 test-coupling sites that would have broken simultaneously, blocking the extraction PRs and forcing a "big bang" rewrite of the test suite. The property-bridge policy let the extraction land incrementally; the eventual bridge cleanup is sequenced after the test-pattern cleanup (D1 arc), which is the correct order.
|
||||
@@ -0,0 +1,85 @@
|
||||
# ADR-0002: Capability Protocol pattern (`SessionCapabilities` fat union)
|
||||
|
||||
> **Current state (2026-05-29).** This ADR is **Superseded** and documents a
|
||||
> pre-cutover pattern for historical context only. Since it was written, the
|
||||
> concrete `Session` facade class, `_session.py`, and `_core.py` have all been
|
||||
> **deleted**, and the surviving shared Protocols now live in
|
||||
> `src/notebooklm/_runtime/contracts.py`
|
||||
> (see [ADR-0014](0014-feature-local-runtime-adapters.md) and
|
||||
> [`docs/architecture.md`](../architecture.md) for the live shape). The
|
||||
> feature-local composite Protocols `ChatRuntime` and `ArtifactsRuntime`
|
||||
> referenced in the Status line below were also **retired** — feature APIs now
|
||||
> take their narrow collaborators by keyword-only constructor argument. The
|
||||
> live client also has later-added namespaces such as `mind_maps` and `labels`.
|
||||
> Read
|
||||
> in-body references to `Session`, `_core.py`, `_session_contracts.py`,
|
||||
> `ChatRuntime`, `ArtifactsRuntime`, and exact `file.py:NNN` line numbers as
|
||||
> historical — they do not point at live code.
|
||||
|
||||
## Status
|
||||
|
||||
Superseded by [`arch-d2-cutover`](https://github.com/teng-lin/notebooklm-py/pull/835) (#835). The `SessionCapabilities` adapter and the transitional `ChatStreamingProvider` Protocol were deleted at cutover time; the later composable-capabilities arc continued in [ADR-0013](0013-composable-session-capabilities.md) (#866) and then retired the concrete `Session` facade itself. Current feature APIs depend on direct collaborators and the small shared Protocol set in `src/notebooklm/_runtime/contracts.py` (`Kernel`, `RpcCaller`, `LoopGuard`); single-consumer Protocols stay local to their owners. `NotebookLMClient` wires those collaborators in `src/notebooklm/_client_assembly.py`.
|
||||
|
||||
This ADR documents the pre-cutover pattern for historical context. The "Decision" section below describes the state prior to D2 cutover; the "Alternatives considered" section describes the replacement now adopted.
|
||||
|
||||
## Context
|
||||
|
||||
At the original baseline, `NotebookLMClient` exposed eight namespaced feature APIs (`notebooks`, `sources`, `artifacts`, `chat`, `research`, `notes`, `settings`, `sharing`). The live client has since added namespaces such as `mind_maps` and `labels`; this ADR describes the earlier pattern that shaped the cutover away from a broad `SessionCapabilities` adapter. Each feature API is implemented in its own module (`_notebooks.py`, `_sources.py`, etc.) and needed structured access to `Session` collaborators (RPC dispatch, auth routing, request-id allocation, polling registry, transport bookkeeping, upload concurrency, etc.).
|
||||
|
||||
Two non-negotiable forces shaped the original design:
|
||||
|
||||
- **Sub-clients must not import `Session` directly.** Doing so would create a circular dependency (`Session` imports sub-clients to expose them on `NotebookLMClient.notebooks` etc.; sub-clients importing `Session` would close the loop). Mypy enforces the boundary via `TYPE_CHECKING` gates today.
|
||||
- **Sub-clients should be typeable.** When a sub-client calls `executor.rpc_call(...)`, mypy needs to verify the signature; passing `Any` defeats the type system at exactly the place where method-ID drift would otherwise break silently.
|
||||
|
||||
The codebase resolved both forces with a *capability Protocol* pattern. Ten narrow `Protocol` classes describe individual collaborator surfaces:
|
||||
|
||||
```text
|
||||
CoreRPCProvider · SourceListProvider · CoreReqIdProvider
|
||||
ChatStreamingProvider · PollRegistryProvider · AuthRouteProvider
|
||||
CookieJarProvider · TransportOperationProvider
|
||||
UploadConcurrencyProvider · LoopAffinityProvider
|
||||
```
|
||||
|
||||
Each Protocol describes the smallest collaborator surface the audit could identify at the time of extraction. A concrete adapter class `SessionCapabilities` then multi-inherits *all ten* Protocols and forwards every method to an underlying `Session` instance (`src/notebooklm/_capabilities.py:149-160`). Sub-clients accept a `SessionCapabilities` parameter in their constructors and rely on it for every collaborator interaction.
|
||||
|
||||
An internal architecture audit (disease D2) classified the result as a *fat-union god-interface wearing a Protocol mask*. The Protocols are narrow individually, but every sub-client takes the union, so the *effective* contract sub-clients depend on is the full ten-Protocol surface. The hoped-for narrowing never materialized because the adapter pre-merges them.
|
||||
|
||||
## Decision
|
||||
|
||||
For the current arc (tier-10 baseline), the pattern is:
|
||||
|
||||
1. Define narrow capability `Protocol` classes in `src/notebooklm/_capabilities.py`, one per collaborator surface.
|
||||
2. Define a single concrete adapter `SessionCapabilities` that multi-inherits every Protocol and forwards to `Session`.
|
||||
3. Sub-client constructors accept a `SessionCapabilities` instance, not a `Session` instance.
|
||||
4. `NotebookLMClient` constructs one `SessionCapabilities` adapter at open time and threads it into every sub-client.
|
||||
|
||||
The pattern is *Accepted* today because:
|
||||
|
||||
- It provides a single import path for sub-clients (`from ._capabilities import SessionCapabilities`), avoiding the "every sub-client lists its own Protocol grab-bag" boilerplate.
|
||||
- It guarantees that every Protocol has at least one structural implementer (`Session`, through the adapter), so mypy verifies the contract end-to-end.
|
||||
- It survived the tier-7 thread-safety arc and the tier-8 RPC/VCR arc without churn, which means the surface is empirically stable.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- A single, mypy-verified seam between sub-clients and `Session`.
|
||||
- The capability Protocols documented the collaborator graph in one place, which was useful for comparison. The live graph now lives in [`docs/architecture.md`](../architecture.md).
|
||||
|
||||
**Unwanted (and the reason for the sunset clause):**
|
||||
|
||||
- Every sub-client depends on the *union*, not the subset it actually needs. `NotebooksAPI` and `SettingsAPI` do not need `UploadConcurrencyProvider` or `ChatStreamingProvider`; today they advertise both anyway.
|
||||
- `Session` cannot shrink below ~1,300 lines while the union pins it. Every method named in any Protocol must remain on `Session` (or on the adapter, which delegates to `Session`).
|
||||
- The property-bridge zoo in `_core.py:450-774` exists partly because the union forces `Session` to expose attributes that have been physically relocated into seams — see ADR-0001.
|
||||
- The adapter has begun to leak private internals. `_capabilities.py:230` forwards `_core._begin_transport_post`, an underscore-prefixed method. The narrow-protocol contract has already started to bend into private territory.
|
||||
- `ChatStreamingProvider`'s docstring openly self-describes as transitional: *"Chat-aware error mapping still lives on `Session.query_post` until that is extracted into a chat-owned transport."* The fat union is documented as not-yet-done work.
|
||||
|
||||
The audit's recommendation is to *type each sub-client on its actual capability subset* and delete the `SessionCapabilities` adapter. `Session` will structurally satisfy each narrow Protocol; no runtime adapter is needed because structural sub-typing is enough. That work is sequenced as the D2 cutover (Wave 3 of the architecture-disease-remediation arc).
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Per-sub-client narrow `Protocol` class — chosen replacement for the D2 cutover.** Each sub-client declares its own `Protocol` listing only the collaborator surfaces it actually uses. `Session` is not modified; structural sub-typing means `Session` automatically satisfies each per-sub-client `Protocol`. Effect: `NotebooksAPI` depends on `CoreRPC + AuthRoute` only; the type checker enforces narrowing. Cost: each sub-client owns one extra type definition; ~8 new Protocol classes overall. Was not chosen at the time of the original extraction because the team prioritised "one import path for sub-clients" over "minimum coupling per sub-client"; the audit re-prioritises after observing the long-term coupling cost.
|
||||
- **Constructor injection of individual collaborator dataclasses.** Rejected. Would force every sub-client constructor to take 4–7 typed parameters, and every test to construct that many fakes. The Protocol pattern is strictly more ergonomic and only the *unioned shape* is wrong, not the structural-typing approach.
|
||||
- **Type sub-clients on `Session` directly.** Rejected. Creates the circular-import problem and defeats the layering. The Protocol pattern is the canonical Python answer to "type me on something I cannot import."
|
||||
- **`typing.Protocol` with `runtime_checkable=True` + `isinstance` guards instead of an adapter class.** Rejected. `isinstance` checks against `runtime_checkable` Protocols are slow and not actually safer (they do not inspect method signatures). The cost without benefit was clear at the time.
|
||||
- **Delete `SessionCapabilities` outright in this ADR.** Rejected. Removal must be paired with the per-sub-client Protocol introduction; deleting it first would force every sub-client to type its `core` parameter as `Any` during the migration window, which loses the type-safety benefit the pattern was designed to deliver. The D2 cutover sequences the swap atomically.
|
||||
@@ -0,0 +1,114 @@
|
||||
# ADR-0003: `auth.py` write-through facade (`_AuthFacadeModule`)
|
||||
|
||||
## Status
|
||||
|
||||
**Superseded — closed by [ADR-0014](./0014-feature-local-runtime-adapters.md) (session-decoupling plan Waves 3a + 4 T2.2 + 5).**
|
||||
|
||||
The deferred goal — reducing `auth.py` to an almost-flat re-export module — was
|
||||
completed across three PRs in the session-decoupling plan:
|
||||
|
||||
- **#1066** (Wave 3a / Task 2.1) moved `load_auth_from_storage()` body into
|
||||
`_auth/tokens.py`. `notebooklm.auth.load_auth_from_storage` became a
|
||||
one-line re-export.
|
||||
- **#1070** (Wave 4 T2.2) **inverted** the `_validate_required_cookies()`
|
||||
write-through. Instead of copy-forwarding facade-level rebindings of
|
||||
`MINIMUM_REQUIRED_COOKIES` / `_EXTRACTION_HINT` / `_has_valid_secondary_binding`
|
||||
into `_cookie_policy` (and mirroring `_SECONDARY_BINDING_WARNED` back),
|
||||
`notebooklm.auth._validate_required_cookies` is now identity-equal to
|
||||
`notebooklm._auth.cookie_policy._validate_required_cookies`. Tests that
|
||||
need to rebind policy names now patch `_auth.cookie_policy.X` directly.
|
||||
- **#1055** (pre-plan groundwork) had already moved `AuthTokens` into
|
||||
`_auth/tokens.py`; the `auth.py`-level binding became a re-export.
|
||||
|
||||
**Post-condition (verified by
|
||||
`grep -nE "^(async[[:space:]]+)?def |^class " src/notebooklm/auth.py`):**
|
||||
`auth.py` contains exactly one function body, `async def enumerate_accounts`,
|
||||
which remains to bind `_poke_session` as a default dependency, and zero class
|
||||
definitions. Every other top-level name is a one-line re-export from the
|
||||
relevant `_auth/*` module. The historical write-through machinery is fully
|
||||
retired.
|
||||
|
||||
**Why "Superseded by ADR-0014" rather than "Accepted (completed)":** the
|
||||
original ADR-0003 framing was a *write-through* approach (mirror writes
|
||||
through the facade). ADR-0014's Rule 3 closes the same goal by *inversion*
|
||||
(facades become identity-preserving delegates; rebinding happens on the
|
||||
canonical home). The two approaches are not the same shape, so ADR-0003
|
||||
is correctly marked superseded rather than promoted.
|
||||
|
||||
The rest of this ADR is preserved as the historical record of why the
|
||||
write-through facade existed at all.
|
||||
|
||||
## Context
|
||||
|
||||
Authentication concerns (cookie extraction, header construction, refresh, keepalive, account selection, storage on disk) lived in a single `auth.py` module through tier 7. That module reached ~1,600 lines spanning seven loosely-related concerns. Tier 7 (private-module reorg) split it into a `_auth/` subpackage with ten focused modules:
|
||||
|
||||
```text
|
||||
_auth/paths.py storage paths + filesystem helpers
|
||||
_auth/extraction.py cookie/token extraction from browser sessions
|
||||
_auth/headers.py HTTP header construction
|
||||
_auth/cookies.py cookie maps + _update_cookie_input
|
||||
_auth/cookie_policy.py domain allowlist and policy decisions
|
||||
_auth/account.py account profile + multi-account switching
|
||||
_auth/session.py session-level dataclasses
|
||||
_auth/storage.py profile/state persistence on disk
|
||||
_auth/keepalive.py cookie keepalive + __Secure-1PSIDTS rotation
|
||||
_auth/refresh.py token refresh driver
|
||||
```
|
||||
|
||||
`auth.py` survived the split as a *facade module* that re-exports the public surface (functions, dataclasses, constants) and preserves the `notebooklm.auth.<name>` import path for downstream callers. So far so unremarkable.
|
||||
|
||||
What makes this ADR necessary is the *write-through* behavior. The codebase contains ~152 test sites that patch `auth.py`-level names with `monkeypatch.setattr(notebooklm.auth, "<attr>", fake)` (object-attribute form) or `monkeypatch.setattr("notebooklm.auth.<attr>", fake)` (string-target form). Those names had originally lived inside `auth.py`; after the split they live inside `_auth/storage.py`, `_auth/account.py`, `_auth/keepalive.py`, and `_auth/refresh.py`. The patches would silently do nothing if the facade were a passive re-export, because the *consumers* of those names import them directly from the `_auth/*` modules.
|
||||
|
||||
**[Superseded]** `_AuthFacadeModule` was retired in D1 PR-2; production no longer mirrors writes. Historically, the mitigation (`src/notebooklm/auth.py:288-339`) was `_AuthFacadeModule`, a subclass of `types.ModuleType` that overrode `__setattr__` to *mirror* writes from `notebooklm.auth` into each owning seam:
|
||||
|
||||
```python
|
||||
class _AuthFacadeModule(ModuleType):
|
||||
def __setattr__(self, name: str, value: Any) -> None:
|
||||
super().__setattr__(name, value)
|
||||
if name in _AUTH_STORAGE_FACADE_NAMES:
|
||||
setattr(_auth_storage, name, value)
|
||||
if name in _AUTH_ACCOUNT_FACADE_NAMES:
|
||||
setattr(_auth_account, name, value)
|
||||
if name in _AUTH_KEEPALIVE_FACADE_NAMES:
|
||||
setattr(_auth_keepalive, name, value)
|
||||
if name in _AUTH_REFRESH_FACADE_NAMES:
|
||||
setattr(_auth_refresh, name, value)
|
||||
# …additional cross-module mirror rules for headers, cookies,
|
||||
# cookie_policy, and the _poke_session import alias…
|
||||
```
|
||||
|
||||
The class is installed at module import time with `sys.modules[__name__].__class__ = _AuthFacadeModule`. Four name registries plus two cross-module mirror sets enumerate the names that need write-through; the registries are maintained by hand.
|
||||
|
||||
## Decision
|
||||
|
||||
`auth.py` is a facade module installed under a `types.ModuleType` subclass whose `__setattr__` mirrors writes into the owning `_auth/*` seam modules. Four name registries (`_AUTH_STORAGE_FACADE_NAMES`, `_AUTH_ACCOUNT_FACADE_NAMES`, `_AUTH_KEEPALIVE_FACADE_NAMES`, `_AUTH_REFRESH_FACADE_NAMES`) and two cross-module mirror sets (`_REFRESH_DEP_MIRROR_NAMES`, `_KEEPALIVE_DEP_MIRROR_NAMES`) enumerate the names that must mirror.
|
||||
|
||||
The mechanism is *Accepted* today because:
|
||||
|
||||
- It preserves backward compatibility with every existing test that patches `notebooklm.auth.<name>`. Tier 7 would have stalled if the patches had silently no-op'd.
|
||||
- It is invisible to production callers — read paths are normal `__getattribute__` resolution; only writes (which production never does) take the mirror path.
|
||||
- The four name registries are small and explicit; new names are added only when a test introduces a fresh patch site.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- Tier 7's `auth.py` → `_auth/*` extraction shipped without simultaneously rewriting ~152 test sites. The arc could land incrementally.
|
||||
- Production behavior is identical to a flat re-export module; the facade has no runtime cost beyond a single `isinstance`-style branch on attribute writes (which production never executes).
|
||||
|
||||
**Unwanted (and the reason for the sunset clause):**
|
||||
|
||||
- The facade is a *gravity well* for test patterns. Every time a contributor wants to fake an auth helper for a test, the path of least resistance is `monkeypatch.setattr("notebooklm.auth.X", fake)`. That pattern compounds: each new test site adds to the registry that the facade must maintain.
|
||||
- The four name registries are maintained by hand. When `_auth/storage.py` gains a new function that a test wants to patch, the contributor must remember to add the name to `_AUTH_STORAGE_FACADE_NAMES` *and* re-confirm that no other `_auth/*` module imports the function under its bare name (otherwise the mirror writes only to one of two places).
|
||||
- The `_REFRESH_DEP_MIRROR_NAMES` / `_KEEPALIVE_DEP_MIRROR_NAMES` cross-module mirror sets encode an even subtler invariant — names that are owned by one seam but aliased into another at import time. A reader has to trace the `from … import …` chains to verify the mirror is complete.
|
||||
- The whole apparatus exists to make tests pass under a pattern (`monkeypatch.setattr("notebooklm.auth.X", …)`) that an internal architecture audit (disease D1) wants to retire entirely.
|
||||
|
||||
The retirement path began in the D1 auth-side PR ([#834](https://github.com/teng-lin/notebooklm-py/pull/834)): the monolithic `tests/unit/test_auth.py` was split into concern-aligned files (`test_auth_storage.py`, `test_auth_account.py`, `test_auth_refresh.py` etc.), monkeypatches were migrated to constructor injection, and `_AuthFacadeModule` itself was deleted. ADR-0014's session-decoupling work finished the second half: `AuthTokens` and `load_auth_from_storage()` now live in `_auth/tokens.py`, `_validate_required_cookies` is a direct `_auth.cookie_policy` re-export, and `async def enumerate_accounts` is the only remaining `auth.py` function body (see the **Status** block above for the current contract).
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Constructor injection via factories — chosen replacement for the D1 auth-side PR.** Tests construct fakes by calling a `make_fake_core(**overrides)` factory (or the auth-specific equivalent) and inject them through the public constructor instead of patching module globals. The facade becomes unnecessary because no test reaches into `notebooklm.auth.<name>` anymore. Cost: ~70 test sites in `test_auth.py` plus several dozen scattered elsewhere must be rewritten. The migration is sequenced explicitly so the rewrite lands in one auditable PR.
|
||||
- **Delete `_AuthFacadeModule` outright without migrating tests.** Rejected. The audit measured ~152 object-attribute patches and 58 string-target patches across the test suite, many of them targeting `notebooklm.auth.<name>`. Removing the facade in isolation would break those tests with no actionable diagnostic; contributors would re-add an equivalent mechanism under a different name within a tier or two. (This exact regeneration risk is the reason ADR-0001 / ADR-0002 / ADR-0003 are being written *before* the deletion work — the ADR records the trade-off that prevents the rebuild.)
|
||||
- **Move the mirror logic into a `__getattr__`-on-module mechanism.** Rejected. `__getattr__` at module level cannot intercept *writes*, only fallback reads. The patches in scope are writes (`monkeypatch.setattr(...)`), so a read-side fallback would not solve the problem.
|
||||
- **Keep the original monolithic `auth.py` instead of splitting.** Rejected at the time of tier 7. The seven concerns inside `auth.py` had non-overlapping invariants and non-overlapping change cadences; co-locating them was already paying maintenance interest. The split was correct; the facade is the trailing cost of the split done under a test pattern that should not have been load-bearing.
|
||||
- **Selectively retire the facade names (whittle the registries down).** Rejected. Partial retirement would leave a partial gravity well — easier to grow back than to maintain. The D1 plan is "migrate every site, then delete the whole apparatus in one PR."
|
||||
@@ -0,0 +1,59 @@
|
||||
# ADR-0004: Loop-affinity contract for `NotebookLMClient`
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (retroactive). Documents the contract shipped in the tier-7 thread-safety/concurrency arc; reaffirmed by the seam extractions in tier-8/tier-10 and the later `_runtime/` package split.
|
||||
|
||||
## Context
|
||||
|
||||
`NotebookLMClient` is an `async` client built on `httpx.AsyncClient` and a network of `asyncio` primitives — locks, semaphores, condition variables, queues, and a `Task` keepalive loop. All of those primitives bind to the event loop on which they are first awaited (or, for `asyncio.Lock` constructed without `loop=` on 3.10+, the loop running when they are first acquired).
|
||||
|
||||
Three failure modes appeared during the tier-7 audit:
|
||||
|
||||
1. **Cross-loop reuse.** A client opened on loop A, then awaited on loop B (e.g. a different `asyncio.run` invocation, or a different thread's loop). `asyncio.Lock.acquire()` from loop B on a lock owned by loop A either deadlocks (the wake-up is scheduled on a loop that will never run again) or raises a confusing `RuntimeError`, depending on the primitive.
|
||||
2. **Cross-thread reuse.** Each OS thread has its own default loop. Sharing one `NotebookLMClient` instance across threads guarantees cross-loop reuse because each thread enters its own `asyncio.run`.
|
||||
3. **Multi-tenant `AuthTokens` sharing.** The conversation cache is per-instance for a reason: it keys conversation turns by `conversation_id` and does not include `account_email`, so sharing one client between two tenants would leak conversation IDs/turns between accounts.
|
||||
|
||||
The audit chose the simplest possible contract: *an open `NotebookLMClient` instance is bound to its `open()`-time event loop. Open-client cross-loop, cross-thread, and cross-tenant reuse are unsupported. Closing and reopening the same instance on a new loop is supported; `open()` is the binding moment and resets loop-bound collaborators.*
|
||||
|
||||
The contract is enforced at two layers:
|
||||
|
||||
- `src/notebooklm/_loop_affinity.py` exposes `assert_bound_loop(bound_loop)` which compares the current loop to the captured one and raises `RuntimeError` with an actionable diagnostic if they differ.
|
||||
- `src/notebooklm/_runtime/lifecycle.py::ClientLifecycle.open()` captures the loop with `asyncio.get_running_loop()` and exposes it as `get_bound_loop()`. It propagates the binding into collaborators that own loop-bound primitives.
|
||||
- `src/notebooklm/_runtime/transport.py::RuntimeTransport.perform_authed_post()` calls the injected loop check before it enters the middleware chain and before any loop-bound primitive is touched.
|
||||
|
||||
`ClientLifecycle.get_bound_loop()` returns `None` before `open()` is called, and `assert_bound_loop(None)` is a silent no-op. That keeps fresh test fixtures from being misclassified as cross-loop calls before they have opened a transport. The shared capability-Protocol surface that feature APIs depend on lives in `src/notebooklm/_runtime/contracts.py` (`Kernel`, `RpcCaller`, `LoopGuard`). Single-consumer protocols stay local to their owners, such as `AuthMetadata` in `src/notebooklm/_source/upload.py` and `OperationScopeProvider` in `src/notebooklm/_artifact/polling.py`.
|
||||
|
||||
## Decision
|
||||
|
||||
One `NotebookLMClient` instance is bound to the event loop that ran `open()`. The contract is:
|
||||
|
||||
- **Open-client cross-loop sharing is unsupported.** Re-using an already-open client across `asyncio.run` invocations raises `RuntimeError` on the first authed POST. Close → reopen on a different loop is supported and rebuilds/resets loop-bound primitives.
|
||||
- **Cross-thread sharing is unsupported.** Create one `NotebookLMClient` per thread.
|
||||
- **Cross-tenant sharing is unsupported.** Each `AuthTokens` tenant gets its own `NotebookLMClient` instance — `ChatAPI._cache` is per-instance for exactly this reason.
|
||||
|
||||
The contract is enforced via `assert_bound_loop()` (raises `RuntimeError`) rather than via a defensive lock or a silent fallback. A noisy failure on the first violating call is strictly preferable to a deadlock or a leaked conversation ID.
|
||||
|
||||
`ClientLifecycle.get_bound_loop()` returns `None` before `open()` is called; the affinity helper treats `None` as a silent no-op so test fixtures that construct a client without opening it are not penalised.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- The failure mode is *fast and visible*. A cross-loop reuse fails on the first call, with a stack trace pointing at `assert_bound_loop`, not as a mysterious hang ten minutes later.
|
||||
- The contract is *one sentence long*. New contributors do not need to learn six lifecycle rules — they need to learn one rule and one error message.
|
||||
- Each seam (drain, auth refresh, keepalive, transport) can use plain `asyncio.Lock` / `asyncio.Semaphore` without defensive re-binding logic. The cost of cross-loop safety is paid once at the lifecycle layer, not in each seam.
|
||||
|
||||
**Unwanted:**
|
||||
|
||||
- Callers that *want* multi-loop / multi-thread reuse must construct multiple clients. For test code this is mildly verbose; for production code this is the right design (each loop owns its own connection pool) so the tax is paid in tests only.
|
||||
- The pre-open `None` path exists for fixture ergonomics, so the constructor does not enforce a hard loop invariant. The contract is enforced at async entry points that touch transport or loop-bound primitives.
|
||||
- The contract is *advisory* to multi-process callers. Processes do not share Python objects, so the contract is trivially satisfied across fork/spawn boundaries — but the diagnostic message says "loop", not "loop or process", so a reader of an error report has to know that processes are out of scope.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Multi-loop support — rejected.** Would require every loop-bound primitive (lock, semaphore, transport, task) to be re-keyed per loop, plus a coordination layer to migrate state between loops. The complexity blowup is enormous and the use case (one Python process running multiple event loops on multiple threads, all sharing a single NotebookLM client) is not real for the project's audience.
|
||||
- **Thread-safe synchronous client — rejected.** The library is fundamentally async because the underlying RPC surface is async-friendly and the user-facing patterns (polling artifact generations, streaming chat responses) are async-first. A sync wrapper is feasible but it would duplicate every API; the cost outweighs the niche benefit. Callers who need sync access can `asyncio.run(...)` per call.
|
||||
- **Silent rebind on cross-loop access — rejected.** The naive fix is "if `bound_loop != current_loop`, just rebind". This is precisely the bug that creates cross-loop deadlocks: rebinding the *primitive* does not rebind the wake-up callbacks already queued on the old loop, and the old loop is by definition no longer running. The fast-fail rule is intentional.
|
||||
- **Silent no-op on non-loop-bound primitives — partial alternative, applied.** The codebase distinguishes between primitives that *must* fail across loops (anything `asyncio.*`) and primitives that *can* tolerate cross-loop access (read-only dataclasses, metrics counters). The affinity check is applied at the entry points to the loop-bound primitives, not to every method. This is the chosen middle ground.
|
||||
- **Per-instance event loop construction (the client owns its own loop).** Rejected. Would either spawn a thread per client (bad for resource scaling) or hide the loop ownership from the caller (bad for cancellation semantics). Letting the caller own the loop is the canonical async-Python pattern; the client only binds to whichever loop was running at open time.
|
||||
@@ -0,0 +1,83 @@
|
||||
# ADR-0005: Mutating-RPC idempotency taxonomy
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (retroactive). Originally documented the six-policy
|
||||
classification shipped in tier-9 (B1 foundation + Wave-2 classifications
|
||||
across `b-research-notes`, `b-generation`, `b-sources`, and
|
||||
`b-side-effects`). This ADR is the canonical home for the rationale that
|
||||
previously lived in the now-gitignored tier-9 plan; the registry code in
|
||||
`src/notebooklm/_idempotency.py` references this ADR as ADR-0005.
|
||||
|
||||
Amended on 2026-05-29 to remove the unregistered `CLIENT_TOKEN_DEDUPE`
|
||||
policy and executor token-injection hook. No current `RPCMethod` has a
|
||||
verified client-token slot, so keeping the policy made the registry
|
||||
advertise a dead retry-safety mechanism.
|
||||
|
||||
Amended again on 2026-05-29 after the registry audit was completed:
|
||||
the production `IDEMPOTENCY_REGISTRY` now has an explicit entry for
|
||||
every active `RPCMethod`. `UNCLASSIFIED` remains only as a hand-built
|
||||
placeholder for tests and future development, not as the production
|
||||
classification for read-only RPCs. The live singleton still lives in
|
||||
`src/notebooklm/_idempotency.py`; the default policy table is now populated
|
||||
from `src/notebooklm/_idempotency_policy.py` at import time.
|
||||
|
||||
## Context
|
||||
|
||||
The NotebookLM RPC surface is `batchexecute` over HTTPS, and any mutating call (create, delete, refresh, share, generate, …) is susceptible to a *commit-lost* failure: the server commits the write, then the response is lost in transit. A naive retry produces a duplicate write — a duplicate notebook, a duplicate source, an extra LLM inference, a re-sent invite email — depending on the RPC.
|
||||
|
||||
The transport retry layer runs an inner retry loop for transient 5xx / 429 / network-error failures. That loop is *correct* for read-only RPCs and dangerous for mutating ones. Before the taxonomy existed, the only mitigation was a per-call-site decision (`disable_internal_retries=True`) that did not document *why* an RPC was retry-unsafe, so the decision was easy to lose during refactors.
|
||||
|
||||
Five retry-safety profiles cover every verified NotebookLM RPC shape:
|
||||
|
||||
| Policy | Meaning | Effect on the inner retry loop |
|
||||
|---|---|---|
|
||||
| `UNCLASSIFIED` | Placeholder for hand-built test/future registries; not used by the production registry for active RPCs | Silent, retries enabled (preserves pre-taxonomy behavior) |
|
||||
| `PROBE_THEN_CREATE` | Caller owns a probe loop; transport must not blind-retry | Force-disable inner retries |
|
||||
| `IDEMPOTENT_SET_OP` | Server applies set semantics (delete / rename) | Retries are safe; left enabled |
|
||||
| `AT_LEAST_ONCE_ACCEPTED` | Caller has accepted duplicate-side-effect cost | Retries enabled; rate-limited WARN emitted |
|
||||
| `NON_IDEMPOTENT_NO_RETRY` | No dedupe key, no probe; first failure must surface | Force-disable inner retries |
|
||||
|
||||
The taxonomy and the production registry (`IDEMPOTENCY_REGISTRY` in `_idempotency.py`, seeded by `_idempotency_policy.py`) are consulted by `RpcExecutor` to compute the effective `disable_internal_retries` value. Variants (e.g. `ADD_SOURCE` `"url"` vs `"text"` vs `"drive"`, artifact generation/revise flows, and source-label `UPDATE_LABEL` `"add_sources"` / `"remove_sources"`) carry their own classifications when the wire-shape differs by call-site.
|
||||
|
||||
An internal architecture audit (disease D3) flagged ten references to "Wave 2" in the idempotency policy code whose design rationale lived only in internal planning notes. This ADR is the public home for that rationale; the in-code references now point here.
|
||||
|
||||
## Decision
|
||||
|
||||
Every active `RPCMethod` is registered in `IDEMPOTENCY_REGISTRY` (the singleton in `_idempotency.py`, with defaults declared in `_idempotency_policy.py`) with one of five `IdempotencyPolicy` values, optionally per *operation variant* when the call-site shape differs. The registry is the single source of truth consumed by `RpcExecutor`, and the registry-audit tests fail if a new enum member keeps the default placeholder.
|
||||
|
||||
The classification rules are:
|
||||
|
||||
- **Read-only RPCs** classify as `IDEMPOTENT_SET_OP`; replay is explicitly safe because the RPC does not mutate server state.
|
||||
- **Mutating RPCs with a stable server-side dedupe key** classify as `IDEMPOTENT_SET_OP` (delete / rename / set-state) — retries are explicitly safe.
|
||||
- **Mutating RPCs without a dedupe key but with a probe RPC** classify as `PROBE_THEN_CREATE`; the inner retry loop is force-disabled, and the per-API call site owns a probe-then-create wrapper (see `idempotent_create()` in `_idempotency.py` and per-API uses such as `_notebooks.py`, `_source/add.py`, and `_source/upload.py`).
|
||||
- **Mutating RPCs that produce visible side effects (emails, billing, notifications) and that the caller has explicitly opted into** classify as `AT_LEAST_ONCE_ACCEPTED`; retries are enabled but a rate-limited WARN is emitted so operators can observe the trade-off.
|
||||
- **Mutating RPCs with no dedupe key and no reliable probe** classify as `NON_IDEMPOTENT_NO_RETRY`; the inner retry loop is force-disabled and the first failure surfaces to the caller for manual disambiguation.
|
||||
|
||||
The completed production classifications are recorded in `_idempotency_policy.py` (with the per-RPC rationale captured at the registration site). Future classifications continue to land in the same module without changes to the executor; the registry is intentionally extensible.
|
||||
|
||||
The five-policy axis is *closed*. Adding a sixth policy requires updating this ADR and the executor in lock-step.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- Retry safety is now a *property of the RPC*, not a property of the call site. New call sites inherit the safe behavior without re-deriving it.
|
||||
- The executor's retry logic is small and local; the policy decisions live in the registry where they can be reviewed in isolation.
|
||||
- The taxonomy is small enough (five policies) that a reviewer can hold it in mind during a code review. A sixth policy would push past that threshold and is rejected by design.
|
||||
|
||||
**Unwanted:**
|
||||
|
||||
- The registry is populated at module import time and is effectively immutable. A test that wants to override a classification needs to construct a fresh `IdempotencyRegistry` instance (the contract documents this, but it is friction).
|
||||
- `AT_LEAST_ONCE_ACCEPTED`'s rate-limited WARN log is per-process-state (a module-level dict). Tests that observe the log behavior have to manage state across test cases; the WARN is throttled to one emission per 30 seconds per `(method, variant)` to avoid drowning operators in spam, which means a noisy test environment can suppress emissions that would have fired in production.
|
||||
- The taxonomy is *opinionated about caller behavior*. `AT_LEAST_ONCE_ACCEPTED` says "the caller has accepted at-least-once semantics"; if a future contributor classifies an RPC that way without the caller actually having opted in, the registry will silently green-light duplicate side effects. Reviews of new `AT_LEAST_ONCE_ACCEPTED` classifications need to be careful.
|
||||
- The variant-table fallback (`get_entry(method, variant=v)` on a method with no variant table silently falls back to `(method, None)`; the same call on a method *with* a variant table but for an unknown variant raises) is subtle. The contract is documented in the registry class docstring but is the kind of rule that takes a second read to absorb.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Per-call-site `disable_internal_retries=True` flags, no registry.** Rejected. That is the pre-taxonomy state and the audit measured the cost: every refactor risks dropping a flag, and the rationale for "why is this RPC retry-unsafe" lives nowhere. The registry centralises both the decision and the justification (`notes=...` per entry).
|
||||
- **Per-call annotation on the RPC method ID enum (e.g. `RPCMethod.CREATE_NOTEBOOK.policy`).** Rejected. The RPC enum is the source of truth for *method IDs* and is structured to track Google's wire surface. Coupling it to retry semantics would mix transport policy with protocol identity, and the RPC enum is the kind of file that needs to remain mechanically updatable when Google changes a method ID.
|
||||
- **No taxonomy; rely on `httpx`'s built-in retry policy.** Rejected. `httpx` retries are not aware of *which* RPCs are commit-safe to retry. A blind transport-level retry of `CREATE_NOTE` produces a duplicate note; a blind retry of `DELETE_NOTEBOOK` is safe. The transport cannot know which is which; the taxonomy is necessarily a layer above the transport.
|
||||
- **More than five policies.** Rejected. The audit derived the active policy axis by enumerating verified NotebookLM call-site shapes; every shape the codebase has met to date fits one of the five. Adding a policy without a real call-site need would balloon the cognitive surface for reviewers without a corresponding gain.
|
||||
- **Keep a speculative `CLIENT_TOKEN_DEDUPE` policy.** Rejected. The executor previously had token-injection machinery, but no production registry entry used it and no current RPC has a verified client-token slot. A future verified token-dedupe RPC can reintroduce the policy together with the method registration and focused tests.
|
||||
- **Per-call idempotency annotation as a decorator on the API method.** Rejected. The registry-based approach lets `RpcExecutor` consult the policy without per-call-site bookkeeping, and it survives refactors that move call sites between modules. A decorator approach would have to be re-applied on every move.
|
||||
@@ -0,0 +1,62 @@
|
||||
# ADR-0006: VCR cassette scrubber strategy
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (retroactive). Documents the cassette-scrubbing architecture shipped across the tier-8 RPC/VCR remediation arc (51 PRs, all merged) and the follow-up `audit-ref-scrub-2026-05-17` cleanup.
|
||||
|
||||
## Context
|
||||
|
||||
The integration test suite records live `batchexecute` traffic against Google's NotebookLM endpoints using `vcrpy` cassettes (YAML files under `tests/cassettes/`). Recording real traffic captures real authentication material — `__Secure-1PSID*` cookies, OAuth bearer tokens, `at` CSRF tokens, account IDs, email addresses, Drive file IDs, and other PII that must never land in version control.
|
||||
|
||||
Three classes of leak risk exist:
|
||||
|
||||
1. **Direct credential leaks** — full cookie values, full OAuth tokens, the `at` CSRF token, the `__Secure-1PSIDTS` rotation cookie.
|
||||
2. **Domain-specific identifiers** — account email, Drive file IDs, notebook IDs, source IDs. These are not credentials, but they are PII and they are unique enough to fingerprint a real account.
|
||||
3. **Indirect leaks via headers and URLs** — request headers (`Authorization`, `Cookie`, `x-goog-authuser`), URL query parameters (`authuser`, `at`), and response headers (`Set-Cookie`).
|
||||
|
||||
The codebase ships a layered scrubber:
|
||||
|
||||
- **Cassette patterns** (`tests/cassette_patterns.py`) define `SCRUB_PLACEHOLDERS` — the canonical replacement strings (`<SCRUBBED_COOKIE>`, `<SCRUBBED_BEARER_TOKEN>`, `<SCRUBBED_EMAIL>`, etc.) and the regex set that matches each provider-specific leak class.
|
||||
- **VCR config** (`tests/vcr_config.py`) wires the scrubber into `vcrpy`'s `before_record_request` / `before_record_response` hooks, plus a body scrubber that operates on decoded JSON / urlencoded payloads.
|
||||
- **Sanitizer unit tests** (`tests/unit/test_cassette_sanitizer.py`, `tests/_guardrails/test_cassette_shapes.py`) verify scrubbing on a fixed corpus of synthetic payloads.
|
||||
- **Cassette cleanliness guard** (`tests/scripts/check_cassettes_clean.py`) re-scans cassettes and rejects any payload that contains a known leak pattern. CI runs the strict recursive cassette pass and a `--secrets-only --recursive tests/fixtures` pass; `tests/_guardrails/test_cassettes_clean.py` mirrors those paths under pytest. Contributors can still run the script directly for a fast local check.
|
||||
- **Targeted re-scrub script** (`scripts/rescrub-cassettes.py`, covered by `tests/unit/test_rescrub_cassettes_script.py`) applies the documented one-off avatar URL / double-encoded `authuser` repairs and recomputes chunk prefixes. It is not a generic "run every scrub rule over every cassette" tool; new leak classes should add their own focused repair or recorder-side scrubber.
|
||||
|
||||
The audit-ref-scrub cleanup (2026-05-17) added the `is_clean` filter helper so that *widening* a leak detector beyond its original provider allowlist would not re-flag the canonical placeholders (the placeholders contain provider-name substrings). The fix design is recorded in the Decision and Alternatives sections of this ADR — `SCRUB_PLACEHOLDERS` + `is_clean` filter, not regex negative-lookahead.
|
||||
|
||||
## Decision
|
||||
|
||||
Cassette scrubbing is a multi-stage pipeline with the following invariants:
|
||||
|
||||
1. **All scrubbing happens at recording time**, before the cassette is written to disk. Replay-time scrubbing is rejected — once a leak lands on disk it is already in git history.
|
||||
2. **Replacement strings are canonical** and defined in one place (`SCRUB_PLACEHOLDERS` in `tests/cassette_patterns.py`). A placeholder is *idempotent*: running the scrubber twice produces the same output.
|
||||
3. **Leak-class detection** is provider-aware. The regex set distinguishes Google credential shapes (`__Secure-1PSID*`, `at=…`) from email shapes from Drive-file-ID shapes from notebook-ID shapes; each leak class has its own scrub rule.
|
||||
4. **The `is_clean` filter excludes canonical placeholders** from the leak-detection pass. This prevents the "widening a detector re-flags placeholders" failure mode by construction.
|
||||
5. **The cassette guard (`check_cassettes_clean.py`) is the authoritative truth.** A cassette is considered clean iff this guard accepts it. CI invokes it with `--strict --recursive`, and pytest mirrors that gate. Adding new leak classes means updating both the scrubber *and* the guard's expected-placeholder list in lock-step.
|
||||
6. **Sanitizer unit tests are required**, not optional. Any new scrub rule lands with at least one positive case (the rule fires) and one negative case (the canonical placeholder is left alone).
|
||||
7. **Cassettes that pre-date a new scrub rule must be re-scrubbed** before the new rule merges. The re-scrub script is the canonical mechanism; manual edits are forbidden.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- Credentials and PII never reach the on-disk cassette corpus. Tier 8 closed roughly 51 leak classes across as many PRs.
|
||||
- The scrub pipeline is reviewable in isolation: regex set, replacement strings, hook wiring, guard script, sanitizer tests. Each can be audited as a unit.
|
||||
- Idempotency means a contributor can run the scrubber locally during cassette capture without worrying about double-scrubbing.
|
||||
- The guard catches leaks that survive scrubbing (a leak class the scrubber doesn't yet know about) by treating *any* non-placeholder credential-shaped substring as a violation.
|
||||
|
||||
**Unwanted:**
|
||||
|
||||
- The scrubber must run on `before_record_request` *and* `before_record_response` *and* on the JSON body, which means three call sites. A scrub rule added in only one site is a subtle leak path. The sanitizer unit tests cover this by including request, response, and body cases.
|
||||
- The `is_clean` filter is a band-aid against the placeholder-re-flag failure mode. The fundamental issue is that placeholder strings contain provider-name substrings; widening any provider-aware regex risks re-flagging them. The filter handles this without resorting to regex negative-lookahead (which the audit explicitly rejected as a fix).
|
||||
- The full guard is heavier than a normal unit test because it scans the committed cassette corpus and selected fixtures. Contributors who skip `python tests/scripts/check_cassettes_clean.py --strict --recursive` before pushing may still get their first signal from CI. The guard is fast enough for local use, and `--secrets-only` widens the file types for credential hunting without the full cassette false-positive profile.
|
||||
- Re-scrubbing the corpus when a new leak class is added produces large, mechanical diffs that obscure the human-readable diff in the same PR. The convention is to split: "add scrub rule" and "re-scrub corpus" are two commits, ideally two PRs.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **No scrubbing — record real credentials.** Rejected. Cassettes are committed to the repository; real credentials in a public repo are unrecoverable from a security perspective. The cost of even one leak (rotation, revocation, account audit) dwarfs the maintenance cost of the scrubber.
|
||||
- **Scrub per test (every test owns its own scrubber).** Rejected. The leak classes are global: any cassette can contain any leak shape. Per-test scrubbing means every test re-implements the scrub set, drift is inevitable, and the audit cannot easily verify coverage. Centralised scrub rules + central guard is the only design that scales past a handful of cassettes.
|
||||
- **Use vcrpy's built-in `filter_headers` / `filter_query_parameters`.** Partially applied — they handle the header / query layer. They do not handle the response body, do not handle URL-encoded body fields, and do not handle JSON body fields. The scrubber covers what vcrpy does not.
|
||||
- **Encrypt cassettes at rest, scrub at decryption time.** Rejected. Encryption adds key-management burden and does not solve the underlying problem: contributors run tests with the key, so a misconfigured local run can decrypt and re-record under leaked credentials. Plain-text-but-scrubbed is operationally simpler and the security model is "no credentials ever exist in the file" rather than "credentials exist behind a key we trust everyone with."
|
||||
- **Regex negative-lookahead to skip placeholders in the leak detector.** Rejected (and explicitly captured in memory). Negative-lookahead is fragile, slow, and entangles the placeholder strings with every leak pattern. The `is_clean` filter applied after pattern matching is strictly more maintainable.
|
||||
- **CI-only enforcement (skip the local guard).** Rejected. The CI feedback loop is minutes; the local guard's feedback loop is seconds. The audit (specifically the audit-ref-scrub-2026-05-17 cleanup) measured the cost of catching leaks at CI-only and chose to maintain the local guard for the fast loop.
|
||||
@@ -0,0 +1,93 @@
|
||||
# ADR-0007: Test-monkeypatch policy — constructor-injection via `tests/_fixtures/`
|
||||
|
||||
## Status
|
||||
|
||||
Accepted.
|
||||
|
||||
This ADR shipped in the `arch-d1-fixtures-scaffolding` PR (D1 PR-1). It defines the *forbidden* test patterns going forward. The migration of the existing offenders to constructor seams is **complete**: the meta-lint's file-level allowlist has been **drained to zero** (issue #1376), so the per-file gate is now a global invariant and `tests/_guardrails/test_no_forbidden_monkeypatches.py` asserts `_ALLOWLIST == frozenset()` as a hardening guard against re-growth. The policy and lint stay in force indefinitely; only the *migration* is finished.
|
||||
|
||||
**Shim-retirement milestone complete (issue #1367).** All three named production shims this ADR set out to remove are now gone: `_AuthFacadeModule` (retired by ADR-0014), the `_core.py` property-bridge zoo (retired in the session-shrink arc per ADR-0001), and — as of #1367 ([PR #1374](https://github.com/teng-lin/notebooklm-py/pull/1374)) — the last one, the `cli/session_cmd` patch-surface bridge (`_resolve_paths_helper` precedence chain plus its dual fixture and the pure re-export surface). The earlier "only the `cli/session_cmd` block remains / two of three retired" status is superseded by this entry. With the shims retired *and* the allowlist drained to zero (issue #1376), both halves of the policy — shim retirement and offender migration — are complete, so this ADR is now plain "Accepted".
|
||||
|
||||
**Coverage update (issue #1325):** the original meta-lint policed only the three `monkeypatch.setattr` / direct-`AsyncMock`-assignment forms below and was blind to the `unittest.mock` channel — `mock.patch("notebooklm._private…")` / `patch("notebooklm._private…")` / `patch.object(notebooklm._private…, ...)` — which is where most of the recent growth happened. As of #1325 the lint also flags those string-target patches into *private* `notebooklm._*` paths (forbidden pattern 4 below), and the file-level allowlist gained the pre-existing offenders so the policy now covers both mocking channels. Those offenders have since migrated to constructor seams along with the rest, leaving the allowlist empty (issue #1376).
|
||||
|
||||
**Coverage update (2026-06, test-seam layout-coupling gap):** pattern 4's regex anchors on `notebooklm\._` — the private component must be the *first* path component — so the lint was structurally blind to **deep-leaf** private string targets like `patch("notebooklm.cli.session_cmd._sync_server_language_to_config")`, the exact form the #1481 CLI command-move post-mortem showed silently no-op'ing when a command body relocates. The lint now also flags (5) deep-leaf private string targets (at least one public component before the first `_`-prefixed one — deliberately **disjoint** from pattern 4, which keeps its drained-to-zero allowlist and stays-empty guard) and (6) private-*attribute-name* `patch.object(alias, "_private_attr")` patches, where the object reference is sound but the attribute string pins internal layout (only *full* dunder names — leading and trailing double underscore, e.g. `"__aenter__"` — are exempt; `"__private"`-style names are still flagged). Each new pattern carries its own **baselined** file-level allowlist (`_DEEP_LEAF_ALLOWLIST`, `_PATCH_OBJECT_PRIVATE_ATTR_ALLOWLIST`) holding the offenders measured at gate-landing time; unlike the original allowlist these start populated and drain opportunistically (stale-entry checks force removal of cleaned/deleted files, and no new entries are accepted). In the same change, the *remaining* layout coupling — string-target `patch("notebooklm…")` at **public** leaves, which no forbidden-pattern rule covers — was frozen by a per-file shrink-only population ratchet, `tests/_guardrails/test_string_patch_ratchet.py`: baselined files are pinned at their measured site counts, unbaselined files have a budget of zero, and ceilings only tighten. The two gates cross-reference each other in their remediation text so a violator is steered to the sanctioned seams (constructor injection via `tests/_fixtures/`, or `patch.object` on a *public* attribute of a locally-imported alias) rather than into the other gate's forbidden shape.
|
||||
|
||||
## Context
|
||||
|
||||
An internal architecture audit identifies "test-monkeypatch gravity" as the first of three architectural diseases — a *gravity well* in which the test suite's preferred mocking strategy welds module boundaries shut and forces every seam extraction to ship a write-through facade, a property bridge, or a parallel-implementation invariant.
|
||||
|
||||
The audit's verified counts at HEAD `22355cf` are:
|
||||
|
||||
| Pattern | Count |
|
||||
|---|---|
|
||||
| Total `monkeypatch.setattr` sites under `tests/` | 236 |
|
||||
| String-target patches — `monkeypatch.setattr("notebooklm.X.Y", …)` | 58 |
|
||||
| Object-attribute patches — `monkeypatch.setattr(obj, "attr", …)` | 152 |
|
||||
| Direct attribute assignment — `target.rpc_call = AsyncMock(…)` etc. | 63 |
|
||||
| `tests/unit/test_auth_*.py` split (concern-aligned: `test_auth_storage.py`, `test_auth_account.py`, `test_auth_refresh.py` etc.) | Formerly 4,090 LOC · 70 patches |
|
||||
| `tests/unit/cli/test_session.py` size | 4,431 LOC |
|
||||
|
||||
Each of these patterns shares one root cause: production code is mutated from the outside *after* construction, rather than receiving its collaborators *during* construction. At the time this ADR was written, three artifacts visible in `src/notebooklm/` existed solely to keep that style working as the architecture shifts beneath it. **All three are now retired** (see the Status block); they are described here in their original form for historical context:
|
||||
|
||||
- `_AuthFacadeModule` (`auth.py:288-339`) — a `types.ModuleType` subclass whose `__setattr__` mirrors writes from `notebooklm.auth.<name>` across `_auth/storage`, `_auth/account`, `_auth/keepalive`, `_auth/refresh`, plus header/cookie/policy seams. The shim existed because ~152 patches target the `notebooklm.auth` namespace and would silently no-op if the facade were a passive re-export. **Retired** — see ADR-0003 (closed by ADR-0014).
|
||||
- The `_core.py` property-bridge zoo (lines 450-774, ~324 LOC) — read/write properties that delegate to the seam where the storage actually lives. They existed because `monkeypatch.setattr(core, "_save_lock", fake)` was a load-bearing idiom across dozens of tests. **Retired** in the session-shrink arc — see ADR-0001 and ADR-0002.
|
||||
- The `cli/session_cmd.py` proxy block (lines 141-490, ~350 LOC) — module-level functions that mirror service-layer symbols so `monkeypatch.setattr("notebooklm.cli.session_cmd.X", fake)` reaches the real implementation in `cli/services/login.py`. **Retired** by #1367 ([PR #1374](https://github.com/teng-lin/notebooklm-py/pull/1374)), which removed the `_resolve_paths_helper` precedence chain, its dual fixture, and the pure re-export surface — the last of the three named shims to go.
|
||||
|
||||
Every seam extraction up through tier 10 had to add or extend one of these structures rather than break the patches. The cost compounds: each new refactor that crossed one of these load-bearing boundaries either preserved the shim (cementing it) or broke ~100 tests at once. With all three now retired, that gravity is gone for the named shims, and the follow-on work — draining the meta-lint's file-level allowlist as the older offending tests migrate to constructor injection — is **complete: the allowlist drained to zero** (issue #1376).
|
||||
|
||||
The mitigation is a policy change, not a code change. New tests stop using these patterns; existing ones migrate to constructor injection through factories. With no new offenders entering the codebase, the shims become finite, retire-able artifacts rather than permanent fixtures.
|
||||
|
||||
## Decision
|
||||
|
||||
Going forward, every test that needs to substitute a collaborator on `Session` or any sub-client (`NotebooksAPI`, `SourcesAPI`, `ArtifactsAPI`, `ChatAPI`, `ResearchAPI`, `NotesAPI`, `SettingsAPI`, `SharingAPI`) **must** acquire that collaborator through constructor injection, using the factory substrate in `tests/_fixtures/`:
|
||||
|
||||
```python
|
||||
from tests._fixtures import make_fake_core
|
||||
|
||||
async def test_notebooks_list_returns_payload() -> None:
|
||||
fake = make_fake_core(rpc_call=AsyncMock(return_value=[fake_payload]))
|
||||
api = NotebooksAPI(fake.rpc_executor)
|
||||
result = await api.list()
|
||||
fake.rpc_executor.rpc_call.assert_awaited_once()
|
||||
```
|
||||
|
||||
The following patterns are **forbidden** in new test code and enforced by `tests/_guardrails/test_no_forbidden_monkeypatches.py`:
|
||||
|
||||
1. **String-target monkeypatches into the `notebooklm` namespace** — `monkeypatch.setattr("notebooklm.X.Y", ...)`. These rely on import-string resolution and silently no-op when storage relocates.
|
||||
2. **Object-attribute monkeypatches via the `notebooklm` module** — `monkeypatch.setattr(notebooklm.X, "attr", ...)`. Same failure mode; written slightly differently.
|
||||
3. **Direct AsyncMock attribute assignment to the transport/RPC surface** — `target.rpc_call = AsyncMock(...)`, `target._perform_authed_post = AsyncMock(...)`, `target._begin_transport_post = AsyncMock(...)`, `target._finish_transport_post = AsyncMock(...)`, `target.query_post = AsyncMock(...)`, including chained variants like `self._client._target.rpc_call = AsyncMock(...)`. These mutate a constructed instance instead of injecting a fake at construction.
|
||||
4. **`unittest.mock` string-target patches into private internals** (added in #1325) — `mock.patch("notebooklm._private…")` / `patch("notebooklm._private…")` / `patch.object(notebooklm._private…, ...)`. Same import-string failure mode as (1), routed through `unittest.mock` instead of `monkeypatch`. Scoped to paths whose *first* component is private (`notebooklm._*`); deep-leaf privates are pattern (5), and patches at public facades are out of scope for the forbidden-pattern rules (their population is frozen by the string-patch ratchet instead — see the 2026-06 coverage update in the Status block).
|
||||
5. **Deep-leaf `unittest.mock` string-target patches into private internals** (added with the 2026-06 coverage update) — `patch("notebooklm.<public…>._private…")`, e.g. `patch("notebooklm.cli.session_cmd._sync_server_language_to_config")`. The same failure mode as (4) with the private component behind one or more public ones; disjoint from (4) by construction and carries its own baselined, opportunistically-draining file-level allowlist (`_DEEP_LEAF_ALLOWLIST`).
|
||||
6. **Private-attribute-name `patch.object` via a local alias** (added with the 2026-06 coverage update) — `patch.object(alias, "_private_attr", …)`. The object reference is real, but the attribute *name* string pins internal attribute layout and keeps "passing" against bag-of-attribute fakes after a rename. Only full dunder names (`"__aenter__"` etc.) are exempt — Python-protocol surface, not internal layout. Baselined in `_PATCH_OBJECT_PRIVATE_ATTR_ALLOWLIST`; `patch.object` on a *public* attribute remains the sanctioned seam-alias form.
|
||||
|
||||
`tests/_fixtures/fake_core.py` provides `make_fake_core(**overrides) -> FakeSession`. `FakeSession` is a plain explicit-attribute bag whose defaults cover the current shared runtime Protocols in `src/notebooklm/_runtime/contracts.py` (`RpcCaller`, `LoopGuard`, `Kernel`) plus the single-consumer seams that tests still exercise, such as upload auth metadata, artifact polling scope, drain-hook registration, source-list lookup, and queue-wait recording. The factory exposes `fake.rpc_executor.rpc_call` to mirror production wiring and keeps `fake.rpc_call` as a legacy convenience that points at the same mock. Defaults are benign `AsyncMock`s for the async surface and `MagicMock`s for the sync surface; tests override only the slice they exercise via keyword arguments.
|
||||
|
||||
The choice of `types.SimpleNamespace`-shaped attribute storage (rather than `MagicMock(spec=...)` against the fat union or a Protocol-typed class) keeps the factory's per-attribute assignment explicit, prevents `MagicMock` from auto-vivifying attributes the production code does not actually use, and lets reviewers diff the factory against the narrow Protocols a sub-client structurally requires.
|
||||
|
||||
`tests/_fixtures/conftest.py` exposes exactly two thin pytest fixtures — `fake_core` (default factory call) and `make_fake_core` (the factory itself, for tests that need per-call overrides). The deliberately small fixture surface avoids the failure mode `/grill-me` Q11 flagged: a "full fixture menu" creates one parameterless pytest fixture per per-test override combination, which scales O(N tests × M overrides) and quickly becomes worse than the monkeypatch pattern it replaces.
|
||||
|
||||
The meta-lint (`tests/_guardrails/test_no_forbidden_monkeypatches.py`) runs in `pytest` (no skip markers), scans each test file as a single string (so multi-line `monkeypatch.setattr(\n "notebooklm.X", …)` forms are caught — `\s` already matches newlines in Python's `re` engine), and reports each violation with `(file, line, matched pattern)` for actionable failure messages. Its file-level allowlist started at 49 files (the union of audit-flagged offenders at PR-start) and was driven toward zero as offenders migrated. **The allowlist drained to zero** (issue #1376): issue #1325 added a second batch of pre-existing offenders when the lint's coverage was extended to the `unittest.mock` channel (forbidden pattern 4) — files that patch private `notebooklm._*` paths via `mock.patch` / `patch` string targets, concentrated in `notebooklm._research`, `notebooklm._artifact.downloads`, and `notebooklm._sources` — and those migrated to constructor seams along with the rest. The stale-entry guard (a file with no remaining offenders fails the lint) drove the list down without rot, and now that it is empty the lint additionally asserts `_ALLOWLIST == frozenset()` so it cannot re-grow: a new offender must be migrated, never re-allowlisted.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- New tests cannot rebuild the gravity well. The meta-lint blocks the patterns at PR review.
|
||||
- Sub-clients become testable in isolation. `NotebooksAPI(core=fake)` exercises the sub-client without standing up a real `Session`, without import-string resolution, and without after-construction mutation.
|
||||
- The `_AuthFacadeModule` shim, the `_core.py` property bridges, and the `cli/session_cmd.py` proxy block were always intended as *removable* artifacts. They kept existing tests passing until those tests migrated; **all three have now been deleted with no remaining test churn** (D2 cutover and D1 PR-2/PR-3 removed the first two; #1367 / [PR #1374](https://github.com/teng-lin/notebooklm-py/pull/1374) removed the `cli/session_cmd` bridge). The file-level allowlist drain is **complete — the allowlist drained to zero** (issue #1376).
|
||||
- Test diffs become smaller and more readable. A test that overrides one collaborator now shows one keyword argument instead of one `monkeypatch.setattr` line per substituted symbol.
|
||||
|
||||
**Unwanted:**
|
||||
|
||||
- Tests that previously relied on string-target patches to substitute imports from modules they don't directly construct (e.g. patching `notebooklm._core.asyncio.sleep` to skip a real delay) need a different mechanism. The migration plan absorbs these on a case-by-case basis. Most either move to constructor injection of a clock-like collaborator or switch from the *string-target* form — `monkeypatch.setattr("notebooklm._auth.refresh.X", …)`, which the lint forbids because string-target patches resolve at import time and silently no-op on relocation — to the *object-attribute* form against a locally-imported seam alias, which the lint accepts because the alias variable provides a real Python object reference rather than a string the lint cannot validate. The seam-aliased form is the recommended migration target for legitimate `unittest.mock.patch`-style needs that don't fit the constructor-injection model.
|
||||
- The factory carries a small amount of duplication with the narrow Protocols defined in `src/notebooklm/_runtime/contracts.py` and the remaining single-consumer seams in feature modules. Keeping the factory aligned with those Protocols is a manual step; the alternative — generating it — would couple test plumbing to source code in ways that defeat the "tests don't pin shape" property the policy is trying to recover.
|
||||
- The initial allowlist is non-trivial (49 files). Migrating them is a large project, scheduled explicitly as D1 PR-2 (auth-side, 6 files) and D1 PR-3 (CLI-side, 7 files), with the residual ~35 files reassessed at PR-3 close.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Pure pytest fixtures (no factory function).** Rejected per `/grill-me` Q11. A fixture-only world requires one `@pytest.fixture` per override combination — `fake_core_with_rpc_returning_x`, `fake_core_with_authuser_42`, `fake_core_raising_chat_error_on_post`, and so on. The combinatorial explosion is worse than the current monkeypatch sprawl. Hybrid factory-first (factory takes `**overrides`, plus two thin fixtures wrapping the no-override and the factory-itself cases) keeps per-test customization at the call site where it's most readable.
|
||||
- **`MagicMock(spec=SessionCapabilities)` instead of an explicit dataclass.** Rejected. `MagicMock(spec=…)` auto-vivifies attributes against the spec, which means: (a) it silently accepts attribute access the sub-client doesn't actually use, defeating the type-safety benefit of the narrow Protocols, and (b) it ties the test factory to the fat-union `SessionCapabilities` *which D2 PR-2 deletes*. A `types.SimpleNamespace`-shaped explicit class survives the D2 cutover and surfaces shape drift as a hard `AttributeError` rather than as a silent auto-vivified `MagicMock`.
|
||||
- **Per-site allowlist entries (one allowlist entry per `(file, line)` pair).** Rejected per `/grill-me` Q12. Line numbers change on every rebase, generating spurious merge conflicts in the allowlist and making the meta-lint's failures look like merge churn rather than real findings. File-level allowlists trade a small amount of precision (we accept "this file still has offenders") for a stable substrate that survives reorderings.
|
||||
- **One big-bang test rewrite (delete the 273 sites in a single PR).** Rejected. The audit measured 4,090 LOC in `test_auth.py` and 4,431 LOC in `cli/test_session.py`; touching both atomically would block every other test-touching PR for the duration of the cutover and offer no incremental verification. The chosen migration sequence (D1 PR-1 ships the substrate; D1 PR-2 migrates auth tests and deletes `_AuthFacadeModule`; D1 PR-3 migrates CLI tests and deletes the proxy block + property bridges) lets each PR's pytest run prove that the *next* shim removal is safe.
|
||||
- **Deferring the lint to a CI-only check (mypy plugin or separate runner).** Rejected. A pytest-resident lint runs on every contributor's local `uv run pytest` and surfaces violations before push, which is where new violations are easiest to fix. A CI-only check would let local development continue to introduce offenders; the round-trip cost (push → CI fail → fix → push) is exactly the friction the policy is trying to remove.
|
||||
- **Why this matters now.** The audit's 273-site gravity-well analysis (`arch-biggest-problem-audit.md` §D1) shows the cost compounds with every refactor: each new seam extraction either ships a new shim or breaks ~100 tests. Deferring the policy means the next architectural arc (tier 12+ or a hypothetical greenfield split) pays the same tax at higher magnitude. Establishing the substrate now — when the shims that need retirement are still finite and named — keeps the eventual migration bounded.
|
||||
@@ -0,0 +1,87 @@
|
||||
# ADR-0008: `cli/services/` extraction pattern
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (retroactive). Documents the pattern established during the cli-ux-remediation arc (8 phases / 27 PRs, completed 2026-05-14/15) and reinforced by the tier-10 foundation-decomposition work.
|
||||
|
||||
## Context
|
||||
|
||||
CLI commands grew organically: every new feature added a new Click command (or sub-command), and the command body co-located argument parsing, validation, business logic, error handling, and presentation. By the time of the cli-ux audit, the largest CLI module (`cli/session.py`, later renamed `cli/session_cmd.py`, the `login` / `use` / `status` / `clear` family) had reached ~1,973 lines with 64 top-level / async functions. Most of those functions were *business logic*: browser-profile enumeration, cookie extraction, multi-account fan-out, profile validation.
|
||||
|
||||
The audit identified three concrete failure modes:
|
||||
|
||||
1. **Click commands were not unit-testable in isolation.** Testing "what happens if browser profile enumeration returns three entries" required either driving the full Click test runner (slow, mixes parsing + business logic + presentation in every assertion) or `monkeypatch.setattr("notebooklm.cli.session_cmd._enumerate_…", fake)` — the test-monkeypatch gravity pattern that ADR-0003 and ADR-0007 describe.
|
||||
2. **Business logic was uncoupled from re-use.** The same browser-profile enumeration logic would have been useful from the Python API, but importing it required reaching past Click decorators into `cli/session.py`. The CLI module became a sink for logic that should have been library-level.
|
||||
3. **CLI commands could not shrink.** Even the trivial commands carried ~50 lines of validation / setup before they could call into the actual work, because the "actual work" was inline.
|
||||
|
||||
The cli-ux-remediation arc moved business logic into a sibling sub-package:
|
||||
|
||||
```text
|
||||
src/notebooklm/cli/services/
|
||||
├── __init__.py
|
||||
├── download.py download workflow planning
|
||||
├── generate.py generation workflow planning
|
||||
├── login/ browser-cookie auth flows + profile enumeration
|
||||
├── research.py research task workflow helpers
|
||||
├── source_listing.py source list rendering data
|
||||
├── source_mutations.py source mutation helpers
|
||||
└── source_research.py source-grounded research helpers
|
||||
```
|
||||
|
||||
Each `cli/services/<name>.py` module exposes CLI-adjacent workflow logic for one domain. The Click commands in `cli/<name>_cmd.py` shrink to thin shells: parse arguments, construct a service-layer plan or call, render results. Shared user-facing workflows that are no longer CLI-specific now live in `src/notebooklm/_app/` so both CLI, MCP, and server adapters can reuse them without importing Click.
|
||||
|
||||
A typical service module exposes:
|
||||
|
||||
- A `Plan` dataclass that names every decision the command will make (so the plan can be validated / displayed without execution).
|
||||
- A `build_<plan>(args) -> Plan` pure function that validates inputs and constructs the plan.
|
||||
- An `execute_<plan>(plan, facade) -> Result` async function that performs the work, calling out through a `Protocol`-typed facade (so tests can substitute a fake without touching Click).
|
||||
|
||||
The Click command then becomes `parse args → build_plan → execute_plan(plan, real_facade) → render`. Each step is independently testable.
|
||||
|
||||
## Decision
|
||||
|
||||
CLI business logic lives in `src/notebooklm/cli/services/<domain>.py` when it is genuinely CLI-specific, and in `src/notebooklm/_app/<domain>.py` when it is adapter-neutral. Click commands in `src/notebooklm/cli/<domain>_cmd.py` are thin shells that:
|
||||
|
||||
1. Parse arguments via Click decorators.
|
||||
2. Validate inputs (using helpers from the service module where possible).
|
||||
3. Construct a service-layer plan or call the service function directly.
|
||||
4. Render results to the console via the rendering helpers in `cli/rendering.py`.
|
||||
|
||||
Service modules must:
|
||||
|
||||
- Be importable from non-CLI contexts (no top-level Click imports; no `click.echo` calls; no reliance on a Click context).
|
||||
- Define their external collaborators as `Protocol` types (e.g. `SourceAddFacade` in `cli/services/source_add.py`) so tests can pass fakes.
|
||||
- Keep presentation concerns out of the module — return structured results, let the caller decide how to render.
|
||||
- Live next to their consumers — one service module per domain, *not* a `cli/services/utils.py` grab-bag.
|
||||
|
||||
During staged migrations, transitional service modules may appear in the
|
||||
`tests/unit/cli/test_services_boundary.py` inventory with exact documented
|
||||
violations. That inventory is currently empty; every module under
|
||||
`cli/services/` must be classified as guarded or explicitly waived, and new
|
||||
rendering or Click reach-ins fail the boundary tests.
|
||||
|
||||
The pattern is deliberately *light*: there is no service-layer base class, no DI container, no plugin system. Service modules are plain Python modules with plain functions.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- CLI commands shrink toward thin shells. The active command modules are named `*_cmd.py`; `cli/session_cmd.py` is below the historical ≤1,100-line target after the retired patch-surface bridge was removed, and module-size ratchets guard the remaining large CLI modules.
|
||||
- Business logic is unit-testable without driving Click. Tests can call `build_source_add_plan(...)` directly and assert on the returned plan; tests can call `execute_source_add(plan, fake_facade)` and assert on the facade calls.
|
||||
- The service modules document the *contract* of each CLI domain via their `Plan` dataclass and their facade `Protocol`. A reviewer reading `cli/services/source_add.py` sees the entire decision graph for `source add` in one file.
|
||||
- Business logic is re-usable. Adapter-neutral workflows should live under `_app/` (for example `_app/source_clean.py`), while `cli/services/` keeps CLI-only planning and compatibility helpers.
|
||||
- The pattern composes with ADR-0007's test-fixture pattern (constructor injection): service-layer functions take their collaborators as parameters, so test fixtures provide them; no monkeypatching of module globals.
|
||||
|
||||
**Unwanted:**
|
||||
|
||||
- Some Click commands have *no* business logic and still gain a service module if they cross a complexity threshold; reviewers must agree on where that threshold is. The current rule of thumb is "if the command body exceeds 50 lines or has more than one branch on validation results, extract a service."
|
||||
- The pattern *requires* a `Protocol`-typed facade for testability; in some cases the protocol has a single implementer (`NotebookLMClient`) and looks ceremonial. The audit ADR-0002 calls out this exact failure mode for capability Protocols; the mitigation here is that the facade Protocols are narrow (per-service, listing only the methods that service uses) so they do not slip into fat-union shape.
|
||||
- The split is visible in the file count. The login service is now a package of small modules rather than a single `login.py`, and some workflows also have an `_app/` module. The trade is "several reviewable files" vs "one unreviewable file"; the audit chose the former.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Keep business logic inline in `cli/<command>_cmd.py`.** Rejected. The retired proxy block in `cli/session_cmd.py` demonstrated the anti-pattern: the file became a sink of helper functions, monkeypatch surfaces, and "import this for tests" hooks. The lesson is that business logic in CLI modules *will* attract test gravity, and the durable fix is to move the logic out.
|
||||
- **Full `cli/<verb>/<noun>.py` hierarchy** (e.g. `cli/generate/audio.py`, `cli/generate/video.py`). Rejected as overkill for the current size of the surface. The flat `cli/<noun>.py` + `cli/services/<noun>.py` pattern handles the current 9-command surface cleanly; a hierarchical layout would add directory noise without solving a real problem. The pattern is open to revisit if the command count doubles.
|
||||
- **A service-locator / DI container (e.g. `wired`, `dependency-injector`).** Rejected. The codebase has < 10 service modules and < 30 distinct collaborators; the cognitive cost of a DI framework dwarfs the benefit. Plain Python imports + `Protocol`-typed parameters are sufficient at this scale.
|
||||
- **Move business logic into the Python API layer (`_<domain>.py`) and have the CLI call the Python API.** Partial alternative, applied where it fits. The Python API (`NotebooksAPI`, `SourcesAPI`, etc.) carries the *protocol-level* concerns (one RPC = one method). CLI services carry the *workflow-level* concerns (validate user input, choose between two RPC paths, fan out across multiple accounts). The split between "protocol verb" (Python API) and "workflow verb" (CLI service) is real and intentional; not every CLI helper belongs on the Python API.
|
||||
- **Co-locate the service module inside the same file as the command (e.g. `cli/source.py::services`).** Rejected. Single-file co-location was the starting point; the cli-ux audit measured the test friction (every test had to import from a file that also imported Click) and chose physical separation. The cost of one extra import is well below the cost of mixing Click decorators with importable business logic.
|
||||
@@ -0,0 +1,584 @@
|
||||
# ADR-0009: Middleware chain for cross-cutting transport concerns
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (Tier 12 PR 12.1; closed by PR 12.9); context refined by [ADR-0013](0013-composable-session-capabilities.md) (#866).
|
||||
Amended (arc-1, closes the `[arc-1-formalized-as-deferred-permanent]`
|
||||
item on the v0.6 architecture-deepening backlog, §6.1): the
|
||||
`RpcRequest.context: dict[str, Any]` shape is the **long-term**
|
||||
chain-metadata carrier — see §"Decision: `RpcRequest.context: dict[str,
|
||||
Any]` is the long-term shape" below for the rationale and the policy
|
||||
governing additions to the vocabulary table.
|
||||
|
||||
This ADR shipped in PR 12.1 of the Tier-12/13 greenfield migration as
|
||||
type-only scaffolding: the Protocol, dataclasses, and `build_chain` helper
|
||||
landed without production wiring. PR 12.2 wired an empty chain into
|
||||
`Session`. PRs 12.3 through 12.8 each extracted one cross-cutting
|
||||
concern into a dedicated middleware. **PR 12.9 closes the tier** — the
|
||||
seven-middleware chain `[Drain, Metrics, Semaphore, Retry, AuthRefresh,
|
||||
ErrorInjection, Tracing]` is fully wired, the leaf
|
||||
is a pure POST, and the underscore-prefixed compatibility aliases were
|
||||
removed. Later architecture cleanup retired the interim authed-transport
|
||||
Adapter; the current terminal path is
|
||||
`MiddlewareChainHost._authed_post_chain_terminal ->
|
||||
RuntimeTransport.terminal -> Kernel.post`. The chain ordering, the
|
||||
`RpcRequest.context` key vocabulary, and the Protocol shape pinned below
|
||||
are the load-bearing contract.
|
||||
|
||||
Two implementation realities diverged from the original PR-12.1 pin and
|
||||
are documented in the "PR 12.9 close-out notes" section at the bottom of
|
||||
this ADR:
|
||||
1. The `AuthRefreshMiddleware` constructor shipped with a simpler shape
|
||||
that defers request-rebuilding to the leaf (`rebuild_headers` /
|
||||
`build_request_factory` closures are NOT yet wired at chain level).
|
||||
2. The RPC concurrency semaphore wraps the chain dispatch (not the leaf),
|
||||
restoring pre-Tier-12 "one slot per logical RPC" semantics.
|
||||
|
||||
The `Kernel.post` terminal revisit has landed. The live
|
||||
`AuthRefreshMiddleware` constructor uses injected callables
|
||||
(`refresh_callable`, `is_auth_error`, `refresh_callback_enabled`,
|
||||
`refresh_retry_delay`, optional `snapshot_provider`) rather than the
|
||||
older closure-callback target described below; that historical target is
|
||||
kept as tier-12 context, not as the current implementation signature.
|
||||
|
||||
ADR-0002 ("Capability Protocol pattern, `SessionCapabilities` fat
|
||||
union") was superseded by the `arch-d2-cutover` PR (D2 PR-2), per
|
||||
ADR-0002's own Status line. ADR-0010 was the original Tier-13 supersession plan but was itself superseded by [ADR-0013](0013-composable-session-capabilities.md) ("Composable Session Capabilities") in v0.5.0. See [`docs/architecture.md`](../architecture.md)
|
||||
for the post-supersession capability-protocol model.
|
||||
|
||||
## Context
|
||||
|
||||
The post-remediation `Session` orchestrates six cross-cutting concerns
|
||||
across every authenticated POST. The "Today" column below describes the
|
||||
pre-Tier-12 state (when ADR-0009 was written, before any chain extraction
|
||||
landed); the "Post-Tier-12" column describes where each concern lives
|
||||
after PR 12.9 closed the tier. `_SyntheticErrorTransport` was deleted by
|
||||
PR 12.9; the chain-layer `ErrorInjectionMiddleware` is the only
|
||||
substitution path going forward.
|
||||
|
||||
| Concern | Pre-Tier-12 | Post-Tier-12 (PR 12.9 → today) |
|
||||
|---|---|---|
|
||||
| In-flight drain tracking | `TransportDrainTracker.begin/end` around the call (`_transport_drain.py`) | `DrainMiddleware` (chain pos 0) |
|
||||
| Metrics emission | `ClientMetrics.on_rpc_event` callbacks woven through the legacy transport POST loop (`_client_metrics.py`) | `MetricsMiddleware` (chain pos 1) |
|
||||
| RPC concurrency gate | `asyncio.Semaphore` inside the legacy transport POST loop | `SemaphoreMiddleware` (chain pos 2) |
|
||||
| Retry on 5xx / 429 | inline loops inside the legacy transport POST loop | `RetryMiddleware` (chain pos 3) |
|
||||
| Auth refresh on 401 | inline branch inside the legacy transport POST loop (`_session_auth.py`) | `AuthRefreshMiddleware` (chain pos 4) |
|
||||
| Synthetic error injection (tests) | `_SyntheticErrorTransport` wraps the httpx client (`_error_injection.py`) — DELETED PR 12.9 | `ErrorInjectionMiddleware` (chain pos 5) |
|
||||
| Per-attempt tracing/logging | scattered `logger.debug` calls inside the retry loop | `TracingMiddleware` (chain pos 6) |
|
||||
|
||||
Before the chain extraction, adding another concern (e.g. an
|
||||
idempotency-routing wrapper for retry safety, ADR-0005) required touching
|
||||
the transport POST loop directly. Each concern's state holder
|
||||
(`TransportDrainTracker` / `ClientMetrics` / etc.) was also threaded into
|
||||
that leaf, which meant: (a) a new concern grew the host Interface, and
|
||||
(b) every change to one concern risked regressing the others because they
|
||||
shared a function body.
|
||||
|
||||
The greenfield design proposed lifting each concern into a composable
|
||||
middleware, leaving `Kernel.post` as a pure-transport function. The chain
|
||||
is the composition substrate.
|
||||
|
||||
Five details that shaped this ADR:
|
||||
|
||||
1. **HTTP-level, not RPC-level.** The chain wraps the transport, not the
|
||||
encoder. Middlewares see already-encoded bytes; encoding and decoding
|
||||
live in `RpcExecutor.rpc_call` (Tier 13). This keeps every middleware
|
||||
agnostic of `batchexecute` framing and makes test fixtures small.
|
||||
2. **Around-style, not before/after pairs.** Each middleware receives a
|
||||
`next_call` and decides whether (and how) to invoke it. The
|
||||
`AuthRefreshMiddleware` needs to *transform the request and retry once*
|
||||
on 401 — that idiom is awkward to express as separate before/after
|
||||
hooks and natural as an around handler.
|
||||
3. **One global chain at Session init, not per-call.** The chain is
|
||||
stateless; per-call metadata travels through `RpcRequest.context`.
|
||||
Tests override the middleware list via constructor injection
|
||||
(`Session(kernel, middlewares=[FakeMetrics(), real_drain])`) — never
|
||||
by monkeypatching (ADR-0007).
|
||||
4. **Idempotency resolution happens *above* the chain.**
|
||||
`RpcExecutor.rpc_call` calls
|
||||
`_idempotency.resolve_effective_disable_internal_retries(...)` and
|
||||
stuffs the resolved bool into `RpcRequest.context["disable_internal_retries"]`
|
||||
before chain entry. The `RetryMiddleware` (PR 12.7) reads the bool; it
|
||||
does not see the `IdempotencyPolicy` enum or know about
|
||||
`operation_variant` routing. Keeps the chain ignorant of the
|
||||
mutating-RPC idempotency registry while preserving its semantics.
|
||||
5. **`AuthRefreshMiddleware` callbacks are pinned here, not in PR 12.8.**
|
||||
The constructor's `rebuild_headers` and `build_request_factory`
|
||||
callable signatures (§"AuthRefreshMiddleware constructor signature")
|
||||
are part of *this* ADR so PR 12.8's implementation has zero degrees of
|
||||
freedom on shape. PRs 12.2–12.7 also depend on those signatures
|
||||
because they shape `RpcRequest.context` and the chain's interaction
|
||||
with `AuthSnapshot`.
|
||||
|
||||
## Decision
|
||||
|
||||
A single around-style middleware chain wraps the authenticated POST
|
||||
transport. The chain is built once at Session init and invoked per
|
||||
request.
|
||||
|
||||
### Middleware Protocol
|
||||
|
||||
```python
|
||||
class Middleware(Protocol):
|
||||
async def __call__(
|
||||
self,
|
||||
request: RpcRequest,
|
||||
next_call: NextCall,
|
||||
) -> RpcResponse: ...
|
||||
|
||||
NextCall = Callable[[RpcRequest], Awaitable[RpcResponse]]
|
||||
```
|
||||
|
||||
`RpcRequest` and `RpcResponse` are HTTP-shape dataclasses (`url: str`,
|
||||
`headers: dict[str, str]`, `body: bytes`, `context: dict[str, Any]`). The
|
||||
chain operates on already-encoded HTTP requests; encoding/decoding lives
|
||||
*above* the chain in `RpcExecutor.rpc_call`.
|
||||
|
||||
### Chain ordering (load-bearing)
|
||||
|
||||
The chain is composed in this exact order (outermost → innermost):
|
||||
|
||||
```text
|
||||
Drain → Metrics → Semaphore → Retry → AuthRefresh → ErrorInjection → Tracing → terminal
|
||||
```
|
||||
|
||||
Where `terminal` is
|
||||
`MiddlewareChainHost._authed_post_chain_terminal -> RuntimeTransport.terminal -> Kernel.post`.
|
||||
|
||||
The leftmost middleware in the sequence becomes the outermost wrapper.
|
||||
`build_chain` enforces this ordering by composing in reverse (last
|
||||
middleware is wrapped first around `terminal`).
|
||||
|
||||
`SemaphoreMiddleware` was inserted at chain position 2 in PR 12.9 (see
|
||||
"PR 12.9 close-out notes" below) after the first cut of the audit-find
|
||||
moved the `max_concurrent_rpcs` slot to `Session._perform_authed_post`
|
||||
(outside the chain) and codex caught the resulting Drain-admission
|
||||
regression. PR 12.1 originally pinned six middlewares; the chain is seven
|
||||
post-PR-12.9.
|
||||
|
||||
Per-position rationale:
|
||||
|
||||
- **Drain outermost.** Every in-flight call — including ones that haven't
|
||||
reached the transport yet because Semaphore / Retry / AuthRefresh /
|
||||
ErrorInjection haven't released them — must count toward shutdown
|
||||
drain. Putting Drain inside any of those would let a stuck retry (or a
|
||||
queued call waiting for the semaphore) escape the drain accounting.
|
||||
- **Metrics outside Semaphore.** Metrics measure end-to-end timing
|
||||
*including* the time a call spent waiting for the `max_concurrent_rpcs`
|
||||
slot. (`ClientMetrics` also tracks `rpc_queue_wait_seconds_total`
|
||||
separately via the `RPC_QUEUE_WAIT_CONTEXT_KEY` plumbing — that's just
|
||||
queue time, while Metrics latency covers queue + work.) Placing Metrics
|
||||
inside Semaphore would exclude queue wait from latency, breaking
|
||||
pre-Tier-12 telemetry semantics.
|
||||
- **Semaphore outside Retry.** The `asyncio.Semaphore` is non-reentrant.
|
||||
Placing it inside Retry would let each retry attempt try to acquire a
|
||||
fresh slot, deadlocking under sustained 429s when every slot is held by
|
||||
a retrying call waiting to retry into a slot. Placing it outside Retry
|
||||
bounds the whole retry-and-refresh cohort to one slot per logical RPC
|
||||
(matching the pre-Tier-12 contract).
|
||||
- **Retry outside AuthRefresh.** These are orthogonal failure modes — 5xx
|
||||
/ 429 / network errors trigger `RetryMiddleware`; 401 triggers
|
||||
`AuthRefreshMiddleware`. Nesting prevents infinite-loop duplication
|
||||
(each layer has its own guard). Putting them in the other order would
|
||||
let an auth-refresh-then-success-then-5xx sequence cause a retry that
|
||||
re-triggers the refresh, which the legacy transport loop also guarded
|
||||
against with a per-attempt flag. Both layers honor the same
|
||||
`disable_internal_retries` post-resolution bool: a non-idempotent /
|
||||
probe-then-create write is neither retried on 5xx/429 nor replayed
|
||||
after an auth refresh, because a mid-flight 401/403 can land *after*
|
||||
the server committed the write (issue #1157).
|
||||
- **AuthRefresh outside ErrorInjection.** Test-injected 401s exercise the
|
||||
refresh path realistically — a test that injects a 401 expects the
|
||||
refresh middleware to run, not for the injection to short-circuit
|
||||
before refresh sees it. Putting AuthRefresh inside ErrorInjection would
|
||||
invert that.
|
||||
- **ErrorInjection inside Retry.** Synthetic transient failures should
|
||||
look like network errors to `RetryMiddleware`. Putting ErrorInjection
|
||||
outside Retry would make the retry path invisible to the test,
|
||||
defeating the purpose. Pre-PR-12.6 this was a transport-layer wrapper
|
||||
(`_SyntheticErrorTransport`); PR 12.6 lifted it into the chain and PR
|
||||
12.9 deleted the transport class — substitution is now exclusively a
|
||||
chain-layer concern.
|
||||
- **Tracing innermost.** Tracing logs every actual HTTP attempt, including
|
||||
retried ones. Putting Tracing outside Retry would log only one entry
|
||||
per logical call regardless of retries, losing the per-attempt
|
||||
visibility the original transport debug logging provided.
|
||||
|
||||
### `RpcRequest.context` keys (the chain's metadata vocabulary)
|
||||
|
||||
| Key | Type | Set by | Read by |
|
||||
|---|---|---|---|
|
||||
| `rpc_method` | `str \| None` | `RuntimeTransport.perform_authed_post` (receives the resolved method-name string from `RpcExecutor._execute_once`, which passes `method.name` — never the `RPCMethod` enum itself; chat-side callers pass `None`) | `MetricsMiddleware`, `TracingMiddleware` |
|
||||
| `disable_internal_retries` | `bool` | `RuntimeTransport.perform_authed_post` (receives the post-resolution boolean from `RpcExecutor._execute_once`, which calls `_idempotency.resolve_effective_disable_internal_retries(...)` before invoking the chain) | `RetryMiddleware`, `AuthRefreshMiddleware` (when set, skips the auth-refresh-and-retry replay so a non-idempotent / probe-then-create write is not re-issued after a mid-flight 401/403 — issue #1157) |
|
||||
| `build_request` | `BuildRequest` | `RuntimeTransport.perform_authed_post` (stashed before chain entry as the rebuild recipe) | `AuthRefreshMiddleware._rebuild_request_after_refresh`, `RuntimeTransport.refresh_request_for_current_auth`, `RuntimeTransport.terminal` |
|
||||
| `log_label` | `str` | `RuntimeTransport.perform_authed_post` | `DrainMiddleware`, `RetryMiddleware`, `ErrorInjectionMiddleware`, `AuthRefreshMiddleware`, `TracingMiddleware`, `RuntimeTransport.terminal` |
|
||||
| `auth_snapshot` | `AuthSnapshot` | `RuntimeTransport.perform_authed_post` (initial snapshot before chain entry); refreshed by `AuthRefreshMiddleware._rebuild_request_after_refresh` after a successful refresh, and replaced by `RuntimeTransport.refresh_request_for_current_auth` at the chain leaf when a freshness check detects auth moved while the request was queued | `RuntimeTransport.refresh_request_for_current_auth` (chain-terminal pre-POST freshness check); pair-mutated with the materialized envelope so middlewares never observe a torn `(snapshot, request)` pair |
|
||||
| `auth_refreshed` | `bool` | `AuthRefreshMiddleware` (sets to `True` after a successful refresh, **before** the retry leg) | `AuthRefreshMiddleware` (skip-on-replay guard so a `RetryMiddleware` retry on the post-refresh leg cannot drive a second refresh on a fresh 401) |
|
||||
| `rpc_queue_wait_seconds` | `float` | `SemaphoreMiddleware` (writes queue-wait duration on slot acquire — also exported as `RPC_CONTEXT_RPC_QUEUE_WAIT_SECONDS` from `_middleware/context.py`; `RPC_QUEUE_WAIT_CONTEXT_KEY` remains a compatibility alias in `_middleware/semaphore.py`) | `RuntimeTransport.perform_authed_post` (forwards to `ClientMetrics.record_rpc_queue_wait` after the chain returns) |
|
||||
| `read_timeout` | `float \| None` | `RuntimeTransport.perform_authed_post` (seeded only when a per-request read timeout is supplied — currently the chat path's `chat_timeout`; absent otherwise so metadata RPCs keep the base read window) | `RuntimeTransport.terminal` (forwards to `Kernel.post(read_timeout=...)`, which widens only the streamed-response `read` slot) |
|
||||
| `max_response_bytes` | `int` | `RuntimeTransport.perform_authed_post` (seeded only when a per-request response cap is supplied — currently the chat path's `chat_response_max_bytes`; absent otherwise so metadata RPCs keep the shared response-size guard) | `RuntimeTransport.terminal` (forwards to `Kernel.post(max_response_bytes=...)`, which passes a per-call cap to the streaming size guard) |
|
||||
| `disable_read_timeout_retries` | `bool` | `RuntimeTransport.perform_authed_post` (seeded `True` by the chat path) | `RetryMiddleware` (re-raises read-side post-transmission failures — `ReadTimeout` / `ReadError` / `RemoteProtocolError`, see `_NON_REPLAYABLE_POST_SEND_ERRORS` — instead of replaying the non-idempotent in-flight chat generation; connect/write/pool stay retryable and 401 auth refresh is unaffected) |
|
||||
| `refresh_budget` | `RefreshBudget` | `RpcExecutor.rpc_call` / `RuntimeTransport.perform_authed_post` when a logical RPC carries a shared refresh allowance | `AuthRefreshMiddleware` (shares one once-per-logical-call refresh allowance with decoded-RPC retry handling; absent callers fall back to `auth_refreshed`) |
|
||||
| `retry_deadline` | `RuntimeDeadline` | `RpcExecutor.rpc_call` / `RuntimeTransport.perform_authed_post` — the logical call's aggregate `RuntimeDeadline` (anchored at T0), seeded so a re-entered chain shares one budget instead of restarting the retry clock (issue #1873) | `RetryMiddleware` (inherits it instead of minting a fresh per-chain deadline, so the 429/5xx budget does not restart across a decode-time auth-refresh retry) and `AuthRefreshMiddleware` (clamps its post-refresh sleep to the remaining budget so a wire-401 refresh cannot re-POST past the deadline); absent callers (chat path) fall back to minting/omitting their own deadline |
|
||||
|
||||
Middlewares are forbidden from inventing new keys without an ADR update.
|
||||
The dict is mutable by reference (deliberately) but read-mostly in
|
||||
practice. See
|
||||
§"Decision: `RpcRequest.context: dict[str, Any]` is the long-term shape"
|
||||
below for the rationale and the policy that governs additions.
|
||||
|
||||
> **Note on `operation_variant`.** Idempotency policy is resolved
|
||||
> **before chain entry** in `RpcExecutor._execute_once()` via
|
||||
> `_idempotency.resolve_effective_disable_internal_retries(...)`; the
|
||||
> resolved boolean is what flows through the chain as
|
||||
> `disable_internal_retries`. The chain itself never needs the
|
||||
> per-request `operation_variant` selector, so it is intentionally
|
||||
> absent from this vocabulary.
|
||||
|
||||
### Decision: `RpcRequest.context: dict[str, Any]` is the long-term shape
|
||||
|
||||
The stringly-keyed `dict[str, Any]` on `RpcRequest.context` is the
|
||||
**permanent** chain-metadata carrier. It will **not** be replaced with a
|
||||
typed dataclass, `TypedDict`, or per-key dataclass field. The
|
||||
bounded-vocabulary table above is the contract; the policy below bounds
|
||||
drift. The corresponding consequence summary lives under the "Unwanted"
|
||||
list in §"Consequences" below; this section is the authoritative
|
||||
rationale.
|
||||
|
||||
Rationale (decision crystallized in the arc-1 architecture-deepening
|
||||
review and flagged on the demotion list as
|
||||
`[arc-1-formalized-as-deferred-permanent]` — formalized here rather
|
||||
than deferred to a future arc):
|
||||
|
||||
- The typed-envelope migration that arrived with PR #1018 (`b856e01`)
|
||||
promoted `RpcRequest.url`, `RpcRequest.headers`, and `RpcRequest.body`
|
||||
to typed fields and introduced `BuildRequestResult` for the rebuild
|
||||
path. The HTTP-shape envelope is now typed end-to-end; `context` is the
|
||||
only remaining `dict[str, Any]` surface on the chain envelope, and that
|
||||
is by design.
|
||||
- Typing the per-key shape (`TypedDict`, `dataclass`, or a discriminated
|
||||
union of metadata records) would force a mechanical refactor of every
|
||||
read site (middlewares, terminal, host helpers) for ~400 LOC of churn
|
||||
whose only payoff is replacing one shape of "the key isn't there at
|
||||
runtime" failure with another: a `KeyError` becomes an
|
||||
`AttributeError`, but neither is caught at write time, and `mypy
|
||||
--strict` already cannot prove anything stronger about a `TypedDict`
|
||||
whose keys are populated across module boundaries by middlewares the
|
||||
type checker does not know are in the chain.
|
||||
- The drift protection a typed dict would buy comes — at a fraction of
|
||||
the cost — from the **bounded-vocabulary table + lint enforcement**.
|
||||
Reviewer attention is the actual gate today; the planned meta-lint
|
||||
(see "Follow-up" below) makes that gate enforceable without per-call
|
||||
type machinery.
|
||||
- This is the same trade ADR-0013's composable-capabilities split makes
|
||||
in the other direction: keep the typed surfaces typed, keep the
|
||||
metadata-bag surface stringly when its consumers are a closed set
|
||||
governed by ADR review.
|
||||
|
||||
**Policy: vocabulary additions require an ADR update.**
|
||||
|
||||
- Any new `context` key — read OR written — by a middleware, the
|
||||
terminal, or a host helper requires an ADR-0009 amendment that adds
|
||||
the key to the vocabulary table above with `Set by` / `Read by`
|
||||
columns populated.
|
||||
- Reuse before invention: if an existing key carries the same semantic
|
||||
(e.g. `disable_internal_retries` already encodes "skip my own retry
|
||||
budget"), reuse it; do not coin a near-synonym.
|
||||
- The ADR-update requirement is per-key, not per-PR — a single PR that
|
||||
adds two related keys still amends the table for both.
|
||||
- Removing a key also requires an amendment (mark the row with a
|
||||
retirement note and the closing PR, e.g. `(retired in #NNNN)`); the
|
||||
table is the audit log.
|
||||
- Local-only ephemera that a single middleware writes and reads inside
|
||||
one `await next_call(request)` boundary, never observed by another
|
||||
middleware or by the terminal, is NOT a context key — keep that state
|
||||
on the middleware instance or in a `contextvars.ContextVar` scoped to
|
||||
the call. Context keys are for cross-middleware contract.
|
||||
|
||||
The vocabulary is also centralized in
|
||||
`_middleware.context.ALLOWED_RPC_CONTEXT_KEYS`, and
|
||||
`tests/_guardrails/test_middleware_context_contract.py` scans production
|
||||
middleware and transport code for non-approved literal context keys.
|
||||
Adding a key must update this table, `_middleware/context.py`, and the
|
||||
guard test in the same PR.
|
||||
|
||||
### AuthRefreshMiddleware constructor signature (historical Tier-13 target)
|
||||
|
||||
The signature pinned in this section was the **target** shape for the
|
||||
post-`Kernel.post` rewrite (Tier-13 row 13.2), but the live implementation
|
||||
settled on a smaller callable-injection constructor. Current code should
|
||||
consult `src/notebooklm/_middleware/auth_refresh.py`; the closure-callback
|
||||
pair below is retained only to explain the tier-12 design path:
|
||||
|
||||
```python
|
||||
class AuthRefreshMiddleware:
|
||||
def __init__(
|
||||
self,
|
||||
coordinator: AuthRefreshCoordinator,
|
||||
rebuild_headers: Callable[[AuthSnapshot], Mapping[str, str]],
|
||||
build_request_factory: Callable[[AuthSnapshot], BuildRequestResult],
|
||||
) -> None: ...
|
||||
|
||||
async def __call__(
|
||||
self,
|
||||
request: RpcRequest,
|
||||
next_call: NextCall,
|
||||
) -> RpcResponse:
|
||||
try:
|
||||
return await next_call(request)
|
||||
except (HTTP401, _TransportAuthExpired):
|
||||
# Refresh tokens (coalesced, may raise).
|
||||
await self.coordinator.refresh()
|
||||
# Re-snapshot auth state, rebuild headers + url + body for the retry.
|
||||
# ``build_request_factory`` is called exactly ONCE so the rebuilt
|
||||
# url and body stay consistent (a side-effecting factory must not
|
||||
# produce a torn pair).
|
||||
snapshot: AuthSnapshot = await self.coordinator.snapshot()
|
||||
rebuilt = self.build_request_factory(snapshot)
|
||||
new_headers = dict(self.rebuild_headers(snapshot))
|
||||
if rebuilt.headers is not None:
|
||||
# Per the headers-merge policy below, ``BuildRequestResult.headers``
|
||||
# overlays ``rebuild_headers`` so per-request extras (e.g. an
|
||||
# explicit ``Content-Type`` for an upload variant) win over the
|
||||
# snapshot-derived auth headers.
|
||||
new_headers.update(rebuilt.headers)
|
||||
new_request = dataclasses.replace(
|
||||
request,
|
||||
headers=new_headers,
|
||||
url=rebuilt.url,
|
||||
body=rebuilt.body,
|
||||
)
|
||||
return await next_call(new_request)
|
||||
```
|
||||
|
||||
Pinned details:
|
||||
|
||||
- `coordinator: AuthRefreshCoordinator` — the existing seam from
|
||||
`_session_auth.py:53` (`AuthRefreshCoordinator`). PR 12.8 reuses it.
|
||||
- `rebuild_headers: Callable[[AuthSnapshot], Mapping[str, str]]` — **sync**
|
||||
(no I/O; pure header construction from snapshot). Returns the *full*
|
||||
base header dict for the retry, not a delta. The middleware copies the
|
||||
result into a fresh `dict` via `dict(self.rebuild_headers(snapshot))` so
|
||||
the callback's return is not shared with `RpcRequest.headers`.
|
||||
- `build_request_factory: Callable[[AuthSnapshot], BuildRequestResult]` —
|
||||
**sync**. Returns a `BuildRequestResult` dataclass (`url: str`,
|
||||
`body: bytes`, `headers: Mapping[str, str] | None`) — equivalent to
|
||||
today's `_BuildRequest` tuple return, but as a named dataclass for the
|
||||
new code path. Called **exactly once per retry attempt**: a single
|
||||
invocation produces the rebuilt `url`, `body`, and per-request headers
|
||||
overlay so a side-effecting factory cannot emit a torn `url`/`body`
|
||||
pair.
|
||||
- Headers-merge policy: `rebuild_headers` provides the *base* headers
|
||||
(snapshot-derived auth: CSRF token, session id, X-Goog-AuthUser, etc.).
|
||||
`BuildRequestResult.headers` is an *overlay*: when non-`None`, the
|
||||
middleware merges it on top of the base via `dict.update`. This mirrors
|
||||
the current `materialize_rpc_request(...)` semantics where the `headers`
|
||||
slot in the `BuildRequest` tuple represents per-request extras (e.g. an
|
||||
explicit `Content-Type` for an upload variant) that win over the
|
||||
snapshot defaults. Most call sites today pass `None` here, in which
|
||||
case the base headers from `rebuild_headers` are used unchanged.
|
||||
- Retry semantics: **exactly one** retry per `next_call` invocation. If
|
||||
the retry also raises 401, the exception propagates — no second retry,
|
||||
no recursion. `RetryMiddleware` (outside `AuthRefresh` per the chain
|
||||
ordering) handles non-auth retries.
|
||||
- `AuthSnapshot` and `BuildRequestResult` are promoted from private to
|
||||
public-ish in `_request_types.py` (PR 12.1, alongside `BuildRequest`).
|
||||
The current `_AuthSnapshot` and the tuple-return shape become these
|
||||
named types; the underscore-prefixed originals remain as
|
||||
`__all__`-excluded re-exports for one cycle, then delete in PR 12.9.
|
||||
|
||||
PR 12.8 writes the implementation; everything else (the `Session` wiring
|
||||
of the two callbacks, the retry semantics, the types) is fixed here.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- The transport POST path shrinks to a pure `Kernel.post` terminal after
|
||||
PRs 12.4 (metrics out), 12.5 (drain out), 12.7 (retry out), 12.8 (auth
|
||||
refresh out), and the later Adapter retirement. The terminal has no
|
||||
middleware concerns left.
|
||||
- Each cross-cutting concern becomes independently testable: build a
|
||||
chain with just `[FakeRetry()]` around a terminal stub, drive a failing
|
||||
request, assert the retry happened. No more "the metrics callback fires
|
||||
on the third nested call inside the transport loop if the 429 branch …"
|
||||
tests.
|
||||
- Adding a new concern is a new middleware class plus an entry in the
|
||||
chain ordering — no transport-leaf surgery, no growth of a host
|
||||
Protocol.
|
||||
- The chain ordering becomes a single line of code (`[Drain, Metrics,
|
||||
Semaphore, Retry, AuthRefresh, ErrorInjection, Tracing]`) instead of an
|
||||
implicit invariant scattered across one transport function.
|
||||
|
||||
**Unwanted:**
|
||||
|
||||
- A typed dataclass per request is more allocation than the existing
|
||||
three-tuple from `_BuildRequest`. The cost is in microseconds per RPC;
|
||||
the chain itself does not run in a hot loop (one chain dispatch per
|
||||
authed POST, of which there are at most a few hundred per session).
|
||||
Benchmarks in PR 12.2 will quantify; expected overhead is <1% of
|
||||
per-RPC wall time and dominated by the existing `httpx` POST.
|
||||
- The chain's `context: dict[str, Any]` is dynamically typed by
|
||||
design — see §"Decision: `RpcRequest.context: dict[str, Any]` is the
|
||||
long-term shape" above for the rationale and the bounded-vocabulary
|
||||
policy that holds drift back. The meta-lint originally scoped for PR
|
||||
12.9 was not delivered in that PR; until the follow-up lint noted in
|
||||
the Decision section lands, the bounded-vocabulary table is enforced
|
||||
by reviewer attention on every `request.context["…"]` literal in a
|
||||
diff.
|
||||
- Around-style middlewares can short-circuit (return without calling
|
||||
`next_call`). No production middleware in the Tier-12 set does this,
|
||||
but the protocol allows it; test middlewares that need to (e.g. a
|
||||
"deny all requests" canary) get the capability for free. The lint
|
||||
policy is to grep the production middleware modules for "without
|
||||
calling `next_call`" patterns at review time.
|
||||
- Six small middleware files replace six branches in one large transport
|
||||
function. Total LOC roughly equal; navigation improves (one concern per
|
||||
file) but the call path becomes longer in the stack trace.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Before/after hook pairs (the Flask / Express middleware shape).** Rejected.
|
||||
The `AuthRefreshMiddleware` transforms the request and *retries once*; a
|
||||
before-hook returns the transformed request and an after-hook can observe
|
||||
the response, but neither can re-invoke the chain mid-call. Modelling
|
||||
auth refresh as before/after would require a separate "retry once on
|
||||
401" mechanism inside the transport, defeating the extraction.
|
||||
|
||||
**Single `transport_wrapper: Callable[[NextCall], NextCall]` factory list.**
|
||||
Rejected. The factory shape works for around-style behavior but obscures
|
||||
the request/response types at every wrap site, since each factory has to
|
||||
re-declare its own `async def wrapper(request, /): …` body. The
|
||||
`Middleware: Protocol` shape with `RpcRequest` and `RpcResponse`
|
||||
dataclasses gives type checkers full visibility into the chain at every
|
||||
wrap point.
|
||||
|
||||
**`contextvars.ContextVar` for per-request metadata instead of
|
||||
`RpcRequest.context: dict`.** Rejected. The `dict` is shared by reference
|
||||
across the chain (intentional), is trivially mockable in tests, and shows
|
||||
up plainly in `repr(request)`. `contextvars` would: (a) require every
|
||||
middleware to import the var; (b) be invisible in the request repr; (c)
|
||||
leak state across tests if not cleared. The audit work that goes into
|
||||
`tests/_guardrails/` to enforce `RpcRequest.context` key discipline is cheaper
|
||||
than the audit work to enforce `ContextVar` reset discipline.
|
||||
|
||||
**Build the chain per-call instead of once at Session init.** Rejected.
|
||||
Per-call chain construction would allow per-call middleware lists, which
|
||||
breaks the "the chain is the transport contract" invariant — every call
|
||||
must traverse the same chain, otherwise the chain isn't doing the
|
||||
cross-cutting work it's supposed to. Tests that want different middleware
|
||||
lists construct different `Session` instances.
|
||||
|
||||
**Inline the dataclass definitions in `_middleware.py` and skip
|
||||
`_request_types.py`.** Initially rejected while the transport leaf was still
|
||||
being extracted. The later architecture cleanup accepted the promotion:
|
||||
`AuthSnapshot` and `BuildRequest` are not chain-specific, and owning them in
|
||||
`_request_types.py` keeps `_middleware.py` focused on the chain envelope shape
|
||||
while giving non-chain callers a stable import path.
|
||||
|
||||
**Use `httpx.Request` / `httpx.Response` directly instead of
|
||||
`RpcRequest` / `RpcResponse` dataclasses.** Rejected for the request side;
|
||||
accepted for the response side. The request dataclass needs to carry
|
||||
`context: dict[str, Any]` for chain metadata, and `httpx.Request` has no
|
||||
extension point for that. The response side just carries
|
||||
`httpx.Response` (the field name `RpcResponse.response`) plus context, so
|
||||
the dataclass is a thin wrapper there.
|
||||
|
||||
## PR 12.9 close-out notes
|
||||
|
||||
Two implementation details landed differently than the PR-12.1 pin and
|
||||
are documented here so Tier-13 callers have an authoritative reference.
|
||||
|
||||
### `SemaphoreMiddleware` inserted at chain position 2
|
||||
|
||||
The `max_concurrent_rpcs` slot is acquired by `SemaphoreMiddleware`,
|
||||
which sits between `MetricsMiddleware` and `RetryMiddleware` in the
|
||||
chain. The middleware writes the per-call queue-wait duration to
|
||||
`RPC_CONTEXT_RPC_QUEUE_WAIT_SECONDS` and
|
||||
`RuntimeTransport.perform_authed_post` forwards that value to
|
||||
`ClientMetrics.record_rpc_queue_wait` after the chain returns.
|
||||
|
||||
The placement is constrained by three simultaneous invariants the
|
||||
shipped chain must preserve (codex caught the violations in the first
|
||||
cut of PR 12.9):
|
||||
|
||||
1. **Drain admission scope.** `DrainMiddleware` (chain pos 0) increments
|
||||
`_in_flight_posts` for every call that enters the chain, INCLUDING
|
||||
ones still waiting for the `max_concurrent_rpcs` slot. If the
|
||||
semaphore wait happened OUTSIDE the chain (e.g. wrapping the chain
|
||||
dispatch in `_perform_authed_post`), `client.close()` mid-flight
|
||||
would reject queued tasks instead of waiting for them — a regression
|
||||
vs. the PR-12.5-onwards contract.
|
||||
2. **Metrics latency includes queue wait.** `MetricsMiddleware`
|
||||
(chain pos 1) starts its `perf_counter` BEFORE `next_call` reaches
|
||||
`SemaphoreMiddleware`. Latency emitted on `rpc_latency_seconds_total`
|
||||
and `RpcTelemetryEvent.elapsed_seconds` covers queue wait + work,
|
||||
matching the pre-PR-12.9 (PR 12.8) telemetry shape where Metrics
|
||||
wrapped the leaf-side semaphore.
|
||||
3. **`asyncio.Semaphore` is non-reentrant.** `RetryMiddleware`
|
||||
(chain pos 3) re-invokes its `next_call` on retry attempts. Placing
|
||||
`SemaphoreMiddleware` INSIDE `RetryMiddleware` would have each retry
|
||||
attempt try to acquire a fresh slot, deadlocking under sustained
|
||||
429s when every slot is held by a retrying call waiting to retry
|
||||
into a slot. Placing it OUTSIDE `RetryMiddleware` (chain pos 2)
|
||||
bounds the whole retry-and-refresh cohort to one slot per logical
|
||||
RPC.
|
||||
|
||||
The middleware takes a zero-arg async-context-manager factory rather
|
||||
than a raw `asyncio.Semaphore`, so production wires
|
||||
`ClientComposed.get_rpc_semaphore`. The holder returns a
|
||||
`contextlib.nullcontext` when `max_concurrent_rpcs is None` (unbounded
|
||||
opt-out) — the `async with` collapses to a no-op for that case.
|
||||
|
||||
History: the first cut of PR 12.9 audit-find #1 wrapped the semaphore
|
||||
around `Session._perform_authed_post` directly (outside the chain).
|
||||
Codex caught the Drain-admission regression with a reproducible
|
||||
`max_concurrent_rpcs=1` test case — queued tasks raised `RuntimeError`
|
||||
during shutdown instead of being awaited. `SemaphoreMiddleware`
|
||||
restored the contract while keeping the retry-multi-acquisition guard
|
||||
the original audit-find existed to provide.
|
||||
|
||||
### `AuthRefreshMiddleware` shipped without rebuild closures
|
||||
|
||||
The original §"AuthRefreshMiddleware constructor signature" pinned a
|
||||
shape with two callbacks:
|
||||
|
||||
```python
|
||||
rebuild_headers: Callable[[AuthSnapshot], Mapping[str, str]]
|
||||
build_request_factory: Callable[[AuthSnapshot], BuildRequestResult]
|
||||
```
|
||||
|
||||
PR 12.8 shipped a simpler `AuthRefreshMiddleware` that catches
|
||||
`httpx.HTTPStatusError`, drives the coalesced refresh via
|
||||
`AuthRefreshCoordinator.await_refresh`, marks
|
||||
`RPC_CONTEXT_AUTH_REFRESHED`, and re-invokes `next_call`
|
||||
**with the same `RpcRequest`** — the terminal re-reads the now-refreshed
|
||||
`AuthSnapshot` from the coordinator and rebuilds headers/url/body before
|
||||
calling `Kernel.post`, preserving the pre-Tier-12 semantics.
|
||||
|
||||
Why deferred: lifting `rebuild_headers` and `build_request_factory` into
|
||||
chain-level closures requires the leaf to become a pure POST that accepts
|
||||
already-built bytes/headers (i.e. `Kernel.post` from Tier-13 row 13.2).
|
||||
Doing it before the `Kernel.post` rewrite would create a third
|
||||
request-construction path (chain-side closure + leaf-side rebuild +
|
||||
`_chat_transport.send_authed_post` direct path) that all have to stay in
|
||||
sync — strictly worse than leaving the leaf authoritative for one more
|
||||
tier.
|
||||
|
||||
The `AuthSnapshot` and `BuildRequestResult` named dataclasses landed in
|
||||
PR 12.1 and live in `_request_types.py`. They are unused by
|
||||
`AuthRefreshMiddleware` today but are the target shape for Tier 13.
|
||||
|
||||
Tier-13 follow-up: rewrite
|
||||
`AuthRefreshMiddleware` against the pinned closure-callback signature
|
||||
once `Kernel.post` is the chain leaf. The signature pinned in
|
||||
§"AuthRefreshMiddleware constructor signature" above is the target.
|
||||
|
||||
ADR-0010 (the original target of this forward reference) was itself
|
||||
superseded by ADR-0013 ("Composable Session Capabilities") in v0.5.0.
|
||||
ADR-0009's middleware-chain ordering remains load-bearing; chain
|
||||
construction now lives in `MiddlewareChainBuilder`
|
||||
(`_middleware/chain.py`) — an extraction performed inside this ADR's
|
||||
domain, not a supersession — and the order is preserved by
|
||||
`tests/unit/test_chain_wiring.py`. Status: Accepted (chain order
|
||||
load-bearing).
|
||||
@@ -0,0 +1,107 @@
|
||||
# ADR-0010: Session/Kernel split
|
||||
|
||||
> **Current state (2026-06).** This ADR is **Superseded** (by
|
||||
> [ADR-0013](0013-composable-session-capabilities.md), then
|
||||
> [ADR-0014](0014-feature-local-runtime-adapters.md)) and documents a transient
|
||||
> tier-13 shape for historical context only. Since it was written, the broad
|
||||
> `Session: Protocol` was **deleted**, `_session_contracts.py` was **renamed
|
||||
> to `src/notebooklm/_runtime/contracts.py`**, and the feature-local composite
|
||||
> Protocols `ChatRuntime` and `ArtifactsRuntime` were **retired** (feature APIs
|
||||
> now take their narrow collaborators by keyword-only constructor argument).
|
||||
> The live shared capability Protocols are `Kernel`, `RpcCaller`, and
|
||||
> `LoopGuard` in `_runtime/contracts.py`; single-consumer seams such as
|
||||
> upload auth metadata and artifact polling scope live in their owning feature
|
||||
> modules, and the old `AsyncWorkRuntime` composite was deleted. Drain-hook registration is the
|
||||
> `register_drain_hook(...)` method on `TransportDrainTracker` in
|
||||
> `_transport_drain.py`. Read in-body references to `Session`,
|
||||
> `_session_contracts.py`, `_capabilities.py`, `ChatRuntime`,
|
||||
> `ArtifactsRuntime`, the `DrainHookRegistration` Protocol, and exact
|
||||
> `file.py:NNN` line numbers as historical — consult `CLAUDE.md` and the
|
||||
> current source tree for the live shape.
|
||||
|
||||
## Status
|
||||
|
||||
Superseded by [ADR-0013](0013-composable-session-capabilities.md) (#866).
|
||||
|
||||
Tier-13 stabilised the 5-member Session/3-member Kernel/1-member DrainHookRegistration triad. ADR-0013 documents the post-drift capability-composition model that replaces it.
|
||||
|
||||
## Context
|
||||
|
||||
`Session` currently owns orchestration, RPC encoding/decoding, drain
|
||||
tracking, request-id allocation, cookie access, HTTP lifecycle, and the
|
||||
feature-facing capability surface. The Tier-12 middleware chain isolated
|
||||
cross-cutting transport concerns, but feature APIs still depend on
|
||||
per-feature narrow capability Protocols and shared capability Protocols in `_session_contracts.py`; feature-local runtimes (`ChatRuntime` in `_chat.py:90`, `ArtifactsRuntime` in `_artifacts.py:154`).
|
||||
|
||||
That shape blocks the Tier-13 decomposition: feature APIs need one stable
|
||||
orchestration contract, transport code needs a smaller HTTP-only contract,
|
||||
and Artifacts needs one close-time hook registration affordance without
|
||||
expanding the general Session surface.
|
||||
|
||||
## Decision
|
||||
|
||||
Tier 13 uses three structural contracts in
|
||||
`src/notebooklm/_session_contracts.py`:
|
||||
|
||||
- `Session: Protocol` has exactly five members: `rpc_call`,
|
||||
`transport_post`, `next_reqid`, `assert_bound_loop`, and
|
||||
`operation_scope`.
|
||||
- `Kernel: Protocol` has exactly three members: `post`, `cookies`, and
|
||||
`aclose`.
|
||||
- `DrainHookRegistration: Protocol` has exactly one member:
|
||||
`register_drain_hook`.
|
||||
|
||||
`Session.transport_post` accepts the existing public-ish
|
||||
`BuildRequest` alias from `src/notebooklm/_request_types.py`. New session
|
||||
contracts must not expose `_BuildRequest` in signatures.
|
||||
|
||||
This PR is type-only. It defines the contracts and documentation, but it
|
||||
does not create concrete `_session.py` or `_kernel.py` modules, rename
|
||||
`Session`, move cookies, move `httpx` lifecycle, or wire new runtime
|
||||
behavior.
|
||||
|
||||
Later Tier-13 PRs delete `src/notebooklm/_capabilities.py` and all
|
||||
per-feature `_<X>Core` Protocols after feature APIs are retyped to the
|
||||
new contracts. `DrainHookRegistration` remains separate from `Session` so
|
||||
Artifacts can register close-time polling cleanup without adding
|
||||
feature-specific lifecycle methods to the five-member Session surface.
|
||||
|
||||
## Consequences
|
||||
|
||||
Wanted:
|
||||
|
||||
- Feature APIs converge on one semantic orchestration surface instead of
|
||||
composing many narrow capability fragments.
|
||||
- The transport boundary is small enough for a concrete Kernel to own
|
||||
`httpx.AsyncClient` lifecycle and cookies without also owning RPC
|
||||
orchestration.
|
||||
- Artifacts gets an explicit, typed drain-hook seam while other features
|
||||
remain typed only against `Session`.
|
||||
- The `BuildRequest` alias prevents new Protocol signatures from leaking
|
||||
`_BuildRequest`.
|
||||
|
||||
Unwanted:
|
||||
|
||||
- During the migration window both the old `_capabilities.py` Protocols
|
||||
and the new contracts coexist.
|
||||
- `Session` does not structurally satisfy every new contract in this
|
||||
PR; concrete conformance lands with the later extraction and retyping
|
||||
PRs.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
Keep per-feature `_<X>Core` Protocols. Rejected because the endpoint would
|
||||
still duplicate the same core session operations across feature sub-client
|
||||
modules and keep `_capabilities.py` as a permanent coordination point.
|
||||
|
||||
Add `register_drain_hook` to `Session`. Rejected because it would expand
|
||||
the general feature contract for an Artifacts-only lifecycle need and
|
||||
break the five-member Session gate.
|
||||
|
||||
Expose a `Kernel.stream` member. Rejected because chat receives a fully
|
||||
buffered `httpx.Response`; the streaming byte iteration is an internal
|
||||
transport implementation detail, not a consumer-facing Kernel operation.
|
||||
|
||||
Move concrete classes in this PR. Rejected because PR 13.1 is deliberately
|
||||
type-only; `_session.py` and `_kernel.py` are reserved for later concrete
|
||||
implementation PRs.
|
||||
@@ -0,0 +1,181 @@
|
||||
# ADR-0011: Schema validation policy (strict-decode default)
|
||||
|
||||
> **Current state (2026-06-11).** The strict-default migration described below
|
||||
> completed. The legacy `NOTEBOOKLM_STRICT_DECODE=0` warn-and-return-`None`
|
||||
> soft-mode opt-out was retired in v0.7.0; `rpc/_safe_index.py` is now
|
||||
> strict-only and raises `UnknownRPCMethodError` on schema drift regardless of
|
||||
> the env var.
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (Tier 13 PR 13.9a).
|
||||
|
||||
## Context
|
||||
|
||||
NotebookLM's batchexecute responses are undocumented and obfuscated. Google
|
||||
reshapes them without notice. Every decoder call site in the client walks
|
||||
nested positional lists by integer indices that are pinned only by what we
|
||||
captured in cassettes and observed in production. When Google rotates a
|
||||
shape (a single index shifts by one, a leaf becomes a wrapper, an inner
|
||||
list becomes a dict), the affected call site either: (a) crashes with a
|
||||
raw `IndexError` / `TypeError` from inside a feature module, or (b)
|
||||
silently degrades to whatever the surrounding code happened to do with
|
||||
`None`.
|
||||
|
||||
The Tier-12 remediation introduced
|
||||
`notebooklm.rpc.safe_index` (`src/notebooklm/rpc/_safe_index.py`) as
|
||||
the single shared schema-drift point: callers descend through it by
|
||||
integer indices, and on descent failure the helper raises
|
||||
`notebooklm.exceptions.UnknownRPCMethodError`. PR-12 era introduced the
|
||||
helper with a temporary `NOTEBOOKLM_STRICT_DECODE=0` soft mode so the
|
||||
migration of ~30 call sites from hand-rolled `try/except IndexError`
|
||||
blocks to `safe_index` could land without immediately breaking
|
||||
downstream code that relied on the silently-degrades-to-`None` contract.
|
||||
That soft mode is now retired.
|
||||
|
||||
Two facts shape this ADR:
|
||||
|
||||
1. **The migration is far enough along.** The shared `safe_index` helper
|
||||
is the policy point for the ~30 batchexecute descent sites that
|
||||
migrated off hand-rolled `try/except IndexError` blocks; every
|
||||
migrated site threads a `method_id` / `source` label through. A
|
||||
small number of historical positional decoders remained in feature modules
|
||||
(now under `_artifact/downloads.py`, `_artifact/polling.py`, and `_chat/`)
|
||||
where
|
||||
the parsing logic predates the helper; each guards its own descent
|
||||
with feature-local error recovery, so the strict-default flip does
|
||||
not regress those sites — they will be migrated in Tier 13.x
|
||||
follow-ups. The default switch is no longer gated on the remaining
|
||||
migration work.
|
||||
2. **Soft mode masks real drift.** In soft mode a Google-side shape
|
||||
change produces a `None` return from a feature method (an empty
|
||||
summary, a missing artifact id, an empty `sources` list), which is
|
||||
indistinguishable from a legitimately-empty payload. Operators
|
||||
discover the drift only via downstream consequence (a cron job that
|
||||
reports "no notebooks today") rather than at the decoder boundary
|
||||
where the drift actually happened.
|
||||
|
||||
The two integration test files
|
||||
`tests/integration/test_artifacts_drift.py` and
|
||||
`tests/integration/test_get_summary_drift.py` already pin **both**
|
||||
soft-mode and strict-mode contracts for the representative call sites,
|
||||
which gives us confidence that the strict path is itself
|
||||
well-characterised — flipping the default is a default change, not a
|
||||
behaviour-discovery exercise.
|
||||
|
||||
Three details that shaped this ADR:
|
||||
|
||||
1. **The env-var name was preserved for the runway, then retired.**
|
||||
During the transition, `NOTEBOOKLM_STRICT_DECODE=0` restored soft
|
||||
mode for one release window. Current code ignores the variable for
|
||||
drift policy: `safe_index` raises regardless of its value.
|
||||
2. **The opt-out was explicit and bounded.** Downstream code that was
|
||||
ingesting `None` as a legitimate "decoder couldn't descend" signal
|
||||
had one cycle to adopt `except UnknownRPCMethodError` (an `RPCError`
|
||||
subclass). The soft-mode path was then removed in v0.7.0.
|
||||
3. **The exception remains under `RPCError`.**
|
||||
`UnknownRPCMethodError` is a subclass of `DecodingError` which is a
|
||||
subclass of `RPCError`. Any existing `except RPCError:` handler
|
||||
already covers the strict-mode raise — the new default does not
|
||||
force downstream callers to add a new except-clause unless they are
|
||||
intentionally treating drift as a non-error sentinel.
|
||||
|
||||
## Decision
|
||||
|
||||
`safe_index` raises `UnknownRPCMethodError` on every descent failure.
|
||||
`NOTEBOOKLM_STRICT_DECODE` is ignored for compatibility with old
|
||||
environments that still set it; it no longer enables soft mode.
|
||||
|
||||
### Behavioural contract
|
||||
|
||||
The single decision point is `rpc/_safe_index.py::safe_index`. It does
|
||||
not consult `_env`; it either returns the successfully descended value
|
||||
or raises `UnknownRPCMethodError` with `method_id`, `source`, `path`,
|
||||
and a truncated `data_at_failure` preview.
|
||||
|
||||
### Historical opt-out lifecycle
|
||||
|
||||
- **PR 13.9a:** unset meant strict; explicit `=0` temporarily restored
|
||||
soft mode.
|
||||
- **v0.5.0:** explicit soft-mode fallback logged the drift warning and
|
||||
emitted `DeprecationWarning` before returning `None`.
|
||||
- **v0.7.0:** soft-mode path was removed. `NOTEBOOKLM_STRICT_DECODE`
|
||||
became a no-op for schema-drift policy and no longer affects decode
|
||||
behavior.
|
||||
|
||||
### Test surface
|
||||
|
||||
`tests/unit/test_strict_decode_default.py` is the canonical pin for the
|
||||
retired opt-out. It asserts that unset env and every old truthy/falsy
|
||||
`NOTEBOOKLM_STRICT_DECODE` spelling still result in `safe_index`
|
||||
raising `UnknownRPCMethodError` on drift. `tests/unit/test_safe_index.py`
|
||||
pins the helper's structured diagnostics and import surface.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- Google-side shape changes surface at the decoder boundary as typed
|
||||
exceptions with `method_id`, `path`, `source`, and `data_at_failure`
|
||||
attributes — operators learn about drift before downstream
|
||||
consequences accumulate.
|
||||
- The "the call site returned `None` but I don't know whether the
|
||||
payload was empty or the decoder failed" ambiguity is eliminated:
|
||||
empty payload still produces an empty value at the feature boundary;
|
||||
drift now produces an exception with structured context.
|
||||
- Integration tests that exercise real-shape cassettes implicitly test
|
||||
the strict path (the production default), increasing the value of
|
||||
every cassette playback as a drift canary.
|
||||
- Adding a new feature call site no longer requires the author to
|
||||
reason about "should this default to soft or strict?" — the answer
|
||||
is always strict, and the historical env-var opt-out is documented
|
||||
only to explain why old environments that still set it see no effect.
|
||||
|
||||
**Unwanted:**
|
||||
|
||||
- Downstream code that relied on `None` as a legitimate "shape didn't
|
||||
match" sentinel now sees a raised exception. The migration window has
|
||||
closed; callers should catch `UnknownRPCMethodError` / `DecodingError`
|
||||
/ `RPCError` depending on how broadly they want to handle drift.
|
||||
- Tests that used to exercise soft-mode fallback now assert strict-only
|
||||
behavior or guard legitimate absent optional fields before calling
|
||||
`safe_index`.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Leave the default at `0` indefinitely; document strict mode as a
|
||||
recommended opt-in.** Rejected. The migration is complete; the
|
||||
asymmetry between "the decoder knows the payload doesn't match" and
|
||||
"the caller silently returns an empty value" is the exact failure mode
|
||||
this helper was introduced to eliminate. Leaving the soft default
|
||||
permanent defeats the purpose of the migration that landed in Tier-12
|
||||
and preserves the very ambiguity ADR-0006 (cassette scrubbing) and
|
||||
ADR-0007 (test monkeypatch policy) work to remove from the test suite.
|
||||
|
||||
**Flip the default AND simultaneously delete the soft-mode branch.**
|
||||
Rejected for the original PR. The opt-out gave one release of migration
|
||||
runway for downstream consumers who were relying on the pre-flip
|
||||
contract. The branch was deleted later, after the warning window.
|
||||
|
||||
**Move the toggle to a runtime constructor argument
|
||||
(`NotebookLMClient(strict_decode=True)`) instead of an env var.**
|
||||
Rejected. Decoder strictness is a process-wide deployment decision,
|
||||
not a per-client one. A constructor argument would force every test
|
||||
fixture and every CLI invocation to thread the toggle through, while
|
||||
the env var is set once at the process boundary and reads atomically
|
||||
on each `safe_index` call. The CLI does not need to expose a
|
||||
`--strict-decode` flag either — operators set the env in their shell
|
||||
or systemd unit and the entire process picks up the policy.
|
||||
|
||||
**Make `safe_index` raise unconditionally and delete the env var.**
|
||||
Rejected for the soft-rollout window, but this is the current behavior
|
||||
except that the env var remains a no-op compatibility spelling rather
|
||||
than a process-start failure.
|
||||
|
||||
**Couple this flip to the `__all__` audit (PR 13.9b).** Rejected. The
|
||||
`__all__` audit and migration doc (originally part of t13-9) must
|
||||
wait for t13-8's session/kernel landing to stabilise the public
|
||||
surface; the strict-decode flip has zero overlap with t13-8's code
|
||||
moves and can ship in parallel with waves 4-8 of Tier-13. Splitting
|
||||
t13-9 into t13-9a (this ADR + flip) and t13-9b (`__all__` audit)
|
||||
unblocks the parallel window and shortens the tier's critical path.
|
||||
@@ -0,0 +1,354 @@
|
||||
# ADR-0012: Implementation surface convention (underscore-prefix policy)
|
||||
|
||||
> **Current state (2026-06-11).** The normative body and the illustrative tree
|
||||
> below were written when `_core.py`, `_session.py`, and the `_session_*.py`
|
||||
> seam modules still existed. **`_core.py` and `_session.py` (the concrete
|
||||
> `Session` orchestrator) have since been deleted**, and the former
|
||||
> `_session_*` / `_runtime_*` collaborators now live in the `_runtime/`
|
||||
> package (e.g. `_runtime/config.py`, `_runtime/contracts.py`,
|
||||
> `_runtime/lifecycle.py`, `_runtime/transport.py`) plus `_middleware/`.
|
||||
> The underscore-prefix
|
||||
> *policy* this ADR establishes is unchanged and still in force; only the
|
||||
> example module names are stale. The current module map lives in
|
||||
> [`docs/architecture.md`](../architecture.md). Read in-body
|
||||
> `_core`/`_session`/`_session_*` mentions and exact line numbers as historical.
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (Tier 13 PR 13.9a). Amended by Tier 13 PR 13.8 to reflect
|
||||
the retirement of the lifted `_core_*` modules.
|
||||
|
||||
## Context
|
||||
|
||||
`notebooklm-py` has accumulated a deeply-seamed internal structure: at
|
||||
the time of this ADR, `src/notebooklm/` contains 14 public-named modules
|
||||
(`artifacts.py`, `auth.py`, `client.py`, `config.py`, `exceptions.py`,
|
||||
`io.py`, `log.py`, `migration.py`, `notebooklm_cli.py`, `paths.py`, `research.py`,
|
||||
`types.py`, `urls.py`, `utils.py`) and roughly 50 underscore-prefixed
|
||||
seam modules and subpackages (`_runtime/`, `_middleware/`, `_kernel.py`,
|
||||
`_request_types.py`, `_transport_errors.py`, `_streaming_post.py`,
|
||||
`_rpc_executor.py`, `_artifacts.py`, `_artifact/`, `_chat/`,
|
||||
`_source/`, `_row_adapters/`, the `_auth/` subpackage, etc.).
|
||||
The seam modules carry the bulk of the implementation; the
|
||||
public-named modules are mostly thin re-export facades or lifecycle
|
||||
entry points.
|
||||
|
||||
The convention emerged organically through the Tier-7 / Tier-8 /
|
||||
Tier-11 / Tier-12 / Tier-13 remediations. Each tier extracted one or
|
||||
more concerns out of `_core.py` (or out of a feature module like
|
||||
`_artifacts.py`) into a new `_<scope>_<concern>.py` seam, with a
|
||||
re-export from the parent module preserved for one cycle and then
|
||||
removed. Tier 13 retired the lifted `_core_*` file names in favor of
|
||||
ownership names: `_session_config.py` (knobs), `_session_helpers.py`
|
||||
(pure utilities), `_request_types.py` (request construction),
|
||||
`_transport_errors.py` (transport error mapping), `_streaming_post.py` (HTTP),
|
||||
`_rpc_executor.py`
|
||||
(RPC dispatch), and so on. Later refactors promoted the flat clusters
|
||||
into packages such as `_runtime/`, `_middleware/`, `_artifact/`,
|
||||
`_source/`, and `_chat/`.
|
||||
|
||||
Downstream code, however, sees no documented rule that distinguishes
|
||||
"this module is the public API" from "this module is an implementation
|
||||
seam I should not import from." The `__init__.py` re-exports declare
|
||||
the *intended* public surface, but the seam modules are still
|
||||
directly importable — a `from`-import targeting an underscore-prefixed
|
||||
seam (e.g. importing `RuntimeTransport` directly from `_runtime/transport.py`)
|
||||
succeeds at runtime, and nothing in the package layout signals which
|
||||
imports are safe across releases.
|
||||
|
||||
Three facts shape this ADR:
|
||||
|
||||
1. **The underscore-prefix convention is already universal in this
|
||||
codebase.** Every module that is not a stable, downstream-visible
|
||||
surface starts with `_`. The convention is implicit in the file
|
||||
tree; this ADR makes it explicit and load-bearing.
|
||||
2. **Re-exports are the migration mechanism.** When a seam needs to
|
||||
become public, it is re-exported from a non-underscore-prefixed
|
||||
module (typically `__init__.py`, `auth.py`, `exceptions.py`, or
|
||||
`types.py`); the underscore-prefixed source module stays internal.
|
||||
ADR-0003 (auth facade write-through) and the `_artifacts.py` →
|
||||
`__init__.py` export pattern both rely on this rule.
|
||||
3. **The seam churn is high.** Tier-7 through Tier-13 moved
|
||||
~40 distinct seams; that motion will continue. Pinning down which
|
||||
imports survive cross-release lets the seam authors refactor
|
||||
freely behind the underscore-prefixed boundary without worrying
|
||||
about silent downstream breakage.
|
||||
|
||||
Two implementation details that shaped this ADR:
|
||||
|
||||
1. **`__all__` is the secondary fence.** Per the `__all__` audit
|
||||
landing in PR 13.9b (the t13-9 split-off), every public-named
|
||||
module declares `__all__` listing exactly what it exports. The
|
||||
underscore-prefix convention is the *primary* fence (a module's
|
||||
filename signals stability); `__all__` is the secondary fence
|
||||
inside each public module.
|
||||
2. **The `_auth/` subpackage exists.** Auth has a richer internal
|
||||
structure than other features (`_auth/paths.py`, `_auth/cookies.py`,
|
||||
`_auth/refresh.py`, etc.). Wrapping it in an underscore-prefixed
|
||||
subpackage rather than scattering `_auth_*.py` files at the top
|
||||
level keeps the surface tidy and signals "the entire subpackage is
|
||||
internal — use `auth.py` (the facade)."
|
||||
|
||||
## Decision
|
||||
|
||||
Every Python module in `src/notebooklm/` belongs to exactly one of
|
||||
three categories, signalled by its filename:
|
||||
|
||||
### 1. Public surface (no underscore prefix)
|
||||
|
||||
```text
|
||||
src/notebooklm/
|
||||
├── __init__.py # re-export hub; declares the stable surface
|
||||
├── artifacts.py # public artifact retry helpers
|
||||
├── client.py # NotebookLMClient + lifecycle helpers
|
||||
├── auth.py # auth facade (almost flat re-exports; enumerate_accounts exception)
|
||||
├── types.py # public dataclasses (Notebook, Source, ...)
|
||||
├── exceptions.py # public exception hierarchy
|
||||
├── config.py # process-level configuration helpers
|
||||
├── paths.py # filesystem path resolution helpers
|
||||
├── research.py # public research helpers
|
||||
├── log.py # logging configuration helpers
|
||||
├── io.py # public I/O helpers
|
||||
├── migration.py # on-disk-format migration helpers
|
||||
├── notebooklm_cli.py # Click CLI entry-point assembler
|
||||
├── urls.py # URL helpers
|
||||
└── utils.py # public utility re-exports
|
||||
```
|
||||
|
||||
Public modules are subject to stability guarantees per `docs/stability.md`.
|
||||
Each public module declares `__all__` listing exactly the symbols it
|
||||
exports. Adding or removing a name from a public module's `__all__` is
|
||||
a `MINOR` (or `MAJOR` post-1.0) version bump; renaming/removing a
|
||||
public module itself is a `MAJOR` bump.
|
||||
|
||||
#### Historical-internal subpackages with public-looking names
|
||||
|
||||
Two top-level *subpackages* — `notebooklm.rpc/` and `notebooklm.cli/` —
|
||||
have no underscore prefix but **are not** part of the public surface and
|
||||
are not covered by stability guarantees. They predate this convention
|
||||
and are documented as internal in `docs/stability.md` (RPC internals are
|
||||
internal except the documented `notebooklm.rpc.RPCMethod` import path;
|
||||
the CLI implementation modules are internal — only the `notebooklm`
|
||||
console-script entry point is stable). Renaming them to `_rpc/` and
|
||||
`_cli/` would break documented import paths used by power users and
|
||||
shell scripts, so the names are preserved and their status is asserted
|
||||
in this ADR. New subpackages with the same status (public-looking name
|
||||
but internal contents) MUST be added to this list before merging.
|
||||
|
||||
Newer adapter subpackages `notebooklm.mcp/` and `notebooklm.server/` also
|
||||
have public-looking names because they host console-script adapters, but their
|
||||
Python module internals are not part of the stable Python API; their user-facing
|
||||
contract is the installed command and documented tool/HTTP surface.
|
||||
|
||||
### 2. Implementation seams (single underscore prefix)
|
||||
|
||||
```text
|
||||
src/notebooklm/
|
||||
├── _runtime/ # runtime contracts/config/auth/lifecycle/transport/init
|
||||
├── _middleware/ # middleware chain core, host, and middlewares
|
||||
├── _kernel.py # Kernel concrete transport core
|
||||
├── _request_types.py # authed POST request construction types
|
||||
├── _transport_errors.py # transport exceptions + POST error mapping
|
||||
├── _streaming_post.py # streaming POST helper
|
||||
├── _rpc_executor.py # RPC dispatch executor
|
||||
├── _transport_drain.py # in-flight transport drain state
|
||||
├── _client_metrics.py # metrics state and callbacks
|
||||
├── _reqid_counter.py # chat request-id counter
|
||||
├── _conversation_cache.py # chat conversation cache
|
||||
├── _polling_registry.py # artifact polling registry
|
||||
├── _cookie_persistence.py # cookie save state
|
||||
├── _artifacts.py # ArtifactsAPI implementation
|
||||
├── _artifact/ # per-concern artifact seams
|
||||
├── _chat/ # ChatAPI implementation + chat helpers
|
||||
├── _source/ # per-concern source seams
|
||||
├── _label/ # label payload builders
|
||||
├── _row_adapters/ # strict positional RPC row adapters
|
||||
├── _notebooks.py # NotebooksAPI implementation
|
||||
├── _sources.py, _notes.py, ... # other feature implementations
|
||||
├── _env.py # env-var resolvers (NOTEBOOKLM_*)
|
||||
├── _backoff.py, _atomic_io.py # narrow utility seams
|
||||
└── _auth/ # auth subpackage (entire tree internal)
|
||||
├── __init__.py
|
||||
├── paths.py
|
||||
├── cookies.py
|
||||
├── refresh.py
|
||||
└── …
|
||||
```
|
||||
|
||||
Seam modules are **not** subject to stability guarantees. Their public
|
||||
surface, internal layout, and module location can all change between
|
||||
any two releases. Downstream code that imports from a seam module
|
||||
(for example pulling `RuntimeTransport` out of `_runtime/transport.py`)
|
||||
is using an internal
|
||||
API and accepts the cost of tracking those moves across releases. The
|
||||
library does not promise import-path stability for any name reachable
|
||||
only through an underscore-prefixed module.
|
||||
|
||||
### 3. Test-only helpers (double underscore prefix or under `tests/`)
|
||||
|
||||
Modules whose only consumers are tests live under `tests/` (the
|
||||
canonical home for test fixtures) or, when they must be shipped with
|
||||
the library for typing reasons, are prefixed with `__` and explicitly
|
||||
documented as test-only. No production code in `src/notebooklm/` is
|
||||
permitted to import from a test-only module.
|
||||
|
||||
### Promotion rule
|
||||
|
||||
When a name needs to become public (a new dataclass, a new exception,
|
||||
a new helper), it is **added to a public-named module's `__all__`**
|
||||
(typically `__init__.py` for the package surface, or `auth.py` for an
|
||||
auth-related helper, or `exceptions.py` for a new exception class).
|
||||
The underscore-prefixed seam that *implements* the name stays
|
||||
internal; the public module simply re-exports it:
|
||||
|
||||
```python
|
||||
# src/notebooklm/__init__.py
|
||||
from ._chat import ChatAPI # implementation seam → public re-export
|
||||
|
||||
__all__ = [..., "ChatAPI", ...]
|
||||
```
|
||||
|
||||
`ChatAPI` is now public; `_chat.ChatAPI` remains the canonical source
|
||||
and is free to refactor internally as long as the re-exported symbol
|
||||
keeps its name and signature.
|
||||
|
||||
The reverse demotion (a public name moving to internal) follows the
|
||||
deprecation policy in `docs/stability.md`: one release of
|
||||
`DeprecationWarning` from the public module, then removal in the next
|
||||
minor (0.x) or major (post-1.0) bump.
|
||||
|
||||
<a id="demotion-consolidation-rule"></a>
|
||||
|
||||
### Demotion / consolidation rule
|
||||
|
||||
The promotion rule above governs only the public-surface boundary
|
||||
(seam → public re-export). It says nothing about motion *within* the
|
||||
seam tier — and without a symmetric rule the prefix convention becomes
|
||||
a one-way ratchet toward more files. The Tier-7 through Tier-13 arc
|
||||
biased toward *splitting* seams because every refactor that produced a
|
||||
new `_<domain>_*.py` sibling was self-justifying ("this concern now has
|
||||
a name"), while collapses had no named policy backing them. The result
|
||||
is structural drift: seams that were extracted for a single Tier's
|
||||
purpose accumulate even after their original consumers are gone.
|
||||
|
||||
To balance the ratchet, two seam-tier consolidation triggers apply:
|
||||
|
||||
1. **Sibling fold.** When two `_<domain>_*.py` siblings each have
|
||||
<300 LOC, each imports only the other plus shared deps, **and** no
|
||||
other `notebooklm._*` seam imports either of them by name, they are
|
||||
**candidates to fold** into a single `_<domain>.py` (or
|
||||
`_<domain>_<concern>.py`, picking whichever name remains
|
||||
self-describing).
|
||||
2. **Capability demotion.** When a capability that was previously
|
||||
promoted to a shared Protocol (per
|
||||
[ADR-0013](0013-composable-session-capabilities.md) §1, the ≥2
|
||||
feature-consumer rule) has dropped back to **a single consumer** —
|
||||
i.e. exactly one feature still types against it — that capability
|
||||
is a **candidate to demote** back into its sole consumer as a
|
||||
feature-local Protocol, mirroring the
|
||||
`ChatRuntime` / `ArtifactsRuntime` / `UploadRuntime` pattern in
|
||||
ADR-0013 §3.
|
||||
|
||||
Both triggers are *candidates*, not mandates: a reviewer may decline
|
||||
the fold/demotion (e.g. when the two siblings have meaningfully
|
||||
different audiences, when an imminent third consumer is in flight, or
|
||||
when keeping the shared Protocol clarifies a domain boundary). The
|
||||
rule exists to make consolidation a normal motion that doesn't need a
|
||||
fresh justification each time — not to force collapses against
|
||||
reviewer judgment.
|
||||
|
||||
**Demotion does not require an ADR update.** It is the symmetric
|
||||
counterpart to ADR-0013's promotion rule (also a standing rule that
|
||||
applies per-PR, not per-ADR). A demotion or sibling fold is a normal
|
||||
refactor PR; it references this section as its policy basis, the same
|
||||
way a new shared Protocol references ADR-0013 §1. Only a change to the
|
||||
*criteria themselves* (the <300-LOC threshold, the second-consumer
|
||||
count, or the mutual-isolation condition on cross-seam imports)
|
||||
requires an ADR amendment.
|
||||
|
||||
This rule is paired with the underscore-prefix convention rather than
|
||||
with ADR-0013 because the rule's *evidence* is module-level
|
||||
(file sizes, file-to-file imports inside `src/notebooklm/`) and the
|
||||
rule's *target* is the seam-tier file layout this ADR pins. The two
|
||||
ADRs read as a paired policy: ADR-0013 §1 governs when a
|
||||
single-consumer capability becomes shared; this section governs when
|
||||
a previously-shared capability or an over-decomposed file pair
|
||||
collapses back.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- Downstream code has one rule to apply: "if the module starts with
|
||||
`_`, do not import from it." Anyone reviewing a downstream
|
||||
integration can check the import list against the
|
||||
filename-prefix rule without needing to consult `__all__` or the
|
||||
stability doc for every name.
|
||||
- Seam authors have full freedom to move, split, merge, or rename
|
||||
internal modules between releases as long as the public-named
|
||||
re-export modules continue to expose the same `__all__`.
|
||||
- New ADRs and refactor plans can reference the rule by name ("the
|
||||
underscore-prefix convention from ADR-0012") instead of restating
|
||||
it each time, shortening planning artifacts and reducing ambiguity
|
||||
across the Tier-13+ refactor arc.
|
||||
- Static analysers (linters, IDEs) can be configured to warn on
|
||||
imports from underscore-prefixed modules outside the package
|
||||
itself, giving downstream consumers an automated guardrail.
|
||||
|
||||
**Unwanted:**
|
||||
|
||||
- The convention is not enforced by Python itself. An adversarial
|
||||
consumer can still reach into the `_core` seam directly and the
|
||||
import succeeds. The convention is policy-level, not
|
||||
language-level. The lint helper planned for PR 13.9b will warn on
|
||||
external-callsite imports from underscore-prefixed `notebooklm`
|
||||
submodules but cannot prevent them.
|
||||
- Some seams that *look* public-grade are not. For example,
|
||||
`_runtime/contracts.py`, `_runtime/transport.py`, `_kernel.py`, and
|
||||
`_middleware/chain_host.py` are load-bearing first-party internals,
|
||||
but downstream callers should still use the public facade or typed
|
||||
client API rather than importing those modules directly. If a seam-level
|
||||
name has legitimate downstream demand, promote a deliberate re-export
|
||||
through a public module and pin it in the public API compat manifest.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Drop the prefix convention and rely solely on `__all__`.** Rejected.
|
||||
`__all__` controls only `from foo import *` behaviour; named imports
|
||||
that reach directly into an underscore-prefixed seam (e.g. pulling
|
||||
`RuntimeTransport` out of `_runtime/transport.py` by name) bypass it entirely.
|
||||
Using `__all__` as the sole stability fence would force consumers to
|
||||
consult the docs for every name to learn whether it is stable, and
|
||||
reviewers would have no filename-level signal during code review. The
|
||||
prefix-plus-`__all__` combination is strictly more readable.
|
||||
|
||||
**Make seam modules truly private with package-import tricks (e.g.
|
||||
`importlib` indirection, vendored namespace).** Rejected. Python's
|
||||
import system does not have a real "module-private" concept; emulating
|
||||
it via `importlib` indirection or vendored namespaces would add
|
||||
runtime cost (loader hooks) and obscure stack traces (the
|
||||
auto-generated indirection module shows up in tracebacks). The cost
|
||||
exceeds the benefit for a single-process library.
|
||||
|
||||
**Put all seams under a single `notebooklm._internal` subpackage.**
|
||||
Rejected. The seams are already organised by domain
|
||||
(`_runtime/`, `_artifact/`, `_chat/`, `_source/`, `_middleware/`); collapsing
|
||||
them under a single `_internal/` subpackage would either lose the
|
||||
domain grouping (one flat `_internal/` directory with 50 modules) or
|
||||
duplicate it (`_internal/core/`, `_internal/artifacts/`, ...) for no
|
||||
gain over the existing flat structure. The single-underscore prefix
|
||||
already accomplishes the "this is internal" signal without restructuring.
|
||||
|
||||
**Promote the strict-decode helper to a public name in this PR.**
|
||||
Rejected. ADR-0011 now makes `safe_index` strict-only; downstream code
|
||||
does not need a public `is_strict_decode_enabled()` helper because there
|
||||
is no runtime policy choice to mirror. Exposing such a helper would
|
||||
commit the library to a function-call API surface with no current
|
||||
downstream demand.
|
||||
|
||||
**Tie this ADR to ADR-0010 (Session/Kernel split).** Rejected. ADR-0010
|
||||
pins a specific contract shape (the Session/Kernel contract triad from ADR-0010, now superseded by ADR-0013). This ADR pins a *naming* convention
|
||||
that applies across the entire `src/notebooklm/` tree. The two are
|
||||
complementary: ADR-0010 says *what* the load-bearing contracts are;
|
||||
this ADR says *where* their implementation lives and how downstream
|
||||
code should reference them.
|
||||
@@ -0,0 +1,303 @@
|
||||
# ADR-0013: Composable Session Capabilities and Feature-Local Runtimes
|
||||
|
||||
> **Historical note (2026-06-11).** This ADR is **Accepted** and its core
|
||||
> decision — composable, per-capability Protocols instead of one fat `Session`
|
||||
> contract — remains in force. However, several *names* it cites have changed
|
||||
> since it was written, and some of the feature-local composites it proposed
|
||||
> were later retired. In particular: the contracts module
|
||||
> `_session_contracts.py` was replaced by `_runtime/contracts.py`; the
|
||||
> concrete `Session` facade was **deleted**; and the feature-local composite
|
||||
> Protocols `ChatRuntime`, `ArtifactsRuntime`, and `UploadRuntime` (Decision
|
||||
> §3) were **retired** in favour of feature constructors taking their narrow
|
||||
> collaborators by keyword-only argument (see
|
||||
> [ADR-0014](0014-feature-local-runtime-adapters.md) and the
|
||||
> `_runtime/contracts.py` module docstring). Read the specific module names,
|
||||
> Protocol names, and `file.py:NNN` line numbers below as historical; the
|
||||
> current source tree and [`docs/architecture.md`](../architecture.md) are
|
||||
> authoritative. The live shared capability Protocols today are `Kernel`,
|
||||
> `RpcCaller`, and `LoopGuard` in `_runtime/contracts.py`; `AuthMetadata` is
|
||||
> local to `_source/upload.py`, `OperationScopeProvider` is local to
|
||||
> `_artifact/polling.py`, and `AsyncWorkRuntime` was deleted.
|
||||
|
||||
## Status
|
||||
|
||||
Accepted.
|
||||
|
||||
This ADR ratifies the capability-composition model originally proposed
|
||||
in `docs/refactor-history.md` (revision 5, dated 2026-05-20). It supersedes
|
||||
[ADR-0010](0010-session-kernel-split.md) (Session/Kernel split), which
|
||||
was re-statused to `Superseded by ADR-0013 (#866)` when this ADR landed.
|
||||
|
||||
The 11-step migration originally drafted alongside this ADR landed in
|
||||
full across Phases 1–7 of the capability refactor arc; the broad
|
||||
`Session` protocol was deleted in Phase 7, so this ADR is now a plain
|
||||
`Accepted` record with no outstanding sunset clause.
|
||||
|
||||
## Context
|
||||
|
||||
ADR-0010 (Tier 13 PR 13.1) pinned a deliberately narrow feature-facing
|
||||
`Session: Protocol` with **exactly five members** — `rpc_call`,
|
||||
`transport_post`, `next_reqid`, `assert_bound_loop`, and `operation_scope`
|
||||
— alongside a three-member `Kernel: Protocol` and a one-member
|
||||
`DrainHookRegistration: Protocol`. The intent was that feature APIs would
|
||||
converge on one semantic orchestration contract, while transport stayed
|
||||
isolated and Artifacts received a dedicated drain-hook seam.
|
||||
|
||||
That narrow contract did not hold. At the time this ADR was written, the
|
||||
broad `Session` Protocol in `src/notebooklm/_session_contracts.py` had
|
||||
grown to **eight members**:
|
||||
|
||||
1. `auth` (an `AuthMetadata` property)
|
||||
2. `kernel` (a `Kernel` property)
|
||||
3. `rpc_call(...)`
|
||||
4. `transport_post(...)`
|
||||
5. `next_reqid(...)`
|
||||
6. `assert_bound_loop()`
|
||||
7. `operation_scope(...)`
|
||||
8. `register_drain_hook(...)`
|
||||
|
||||
The five-member intent of ADR-0010 was gone. `auth` and `kernel` had been
|
||||
promoted as members for upload-flow convenience; `register_drain_hook`
|
||||
had been added to the general contract despite a standalone
|
||||
`DrainHookRegistration` Protocol covering the same shape — leaving two
|
||||
redundant protocols carrying the same single-member surface.
|
||||
`transport_post` and `next_reqid` were used by exactly one feature
|
||||
(chat), but every feature that typed against `Session` was coupled to
|
||||
them. (Post-this-ADR, Phase 7 deleted the broad `Session` Protocol
|
||||
entirely; current `_runtime/contracts.py` exposes only `Kernel`,
|
||||
`RpcCaller`, and `LoopGuard`.)
|
||||
|
||||
Re-reading the codebase against ADR-0010's original constraint, the audit
|
||||
identified two categories of capability:
|
||||
|
||||
- **SHARED**: a capability used by ≥2 features today, justifying
|
||||
promotion to a module-level Protocol in `_runtime/contracts.py`.
|
||||
Examples: logical RPC dispatch (`rpc_call`) is used by every feature
|
||||
API; loop-affinity assertion is used by chat plus artifact polling.
|
||||
The concrete `TransportDrainTracker.operation_scope(...)` helper is
|
||||
used by sources upload plus artifact polling, but the named
|
||||
`OperationScopeProvider` Protocol is no longer a shared contract; it
|
||||
lives beside artifact polling, its only Protocol consumer.
|
||||
- **FEATURE-LOCAL**: a capability used by exactly one feature, with no
|
||||
current second consumer. Examples: `transport_post(...)` + chat's
|
||||
manual `next_reqid(...)` bookkeeping (only chat needs them);
|
||||
drain-hook registration (only artifact polling registers a close-time
|
||||
hook today).
|
||||
|
||||
Mixing the two categories into one fat `Session` Protocol forces every
|
||||
feature to declare a structural dependency on capabilities it never
|
||||
calls. It also encourages "promote it just in case" drift: the
|
||||
`auth`/`kernel`/`register_drain_hook` additions happened precisely
|
||||
because there was no convention saying *don't widen the shared contract
|
||||
unless a second consumer exists*.
|
||||
|
||||
Two adjacent service boundaries are entangled in the same shape problem
|
||||
and must move with this ADR:
|
||||
|
||||
- The `_mind_map.py` module currently bundles a generic note-row CRUD
|
||||
service together with a mind-map-specific adapter. The two have
|
||||
different consumers (the mind-map adapter is consumed by Artifacts
|
||||
download paths; the generic note service is consumed by the note
|
||||
generation path), and they need different lifetimes during the
|
||||
migration.
|
||||
- Saving a chat `AskResult` as a note currently lives on `NotesAPI`,
|
||||
but it depends on chat's response shape and chat's conversation cache.
|
||||
This is the wrong direction of dependency — the owner of the data
|
||||
should own the persistence call.
|
||||
|
||||
The pre-existing scaffolding from earlier remediation work makes a
|
||||
careful migration tractable: `NotebookSourceLister`
|
||||
(`_notebook_metadata.py:19`) and `NotebookSourceIdProvider`
|
||||
(`_notebook_metadata.py:26`) already let `NotebooksAPI` accept a
|
||||
collaborator-shaped dependency without round-tripping through `Session`.
|
||||
`_mind_map.py:55` is the existing service boundary the note/mind-map
|
||||
split rides on.
|
||||
|
||||
## Decision
|
||||
|
||||
The library adopts a **composable capability model** with feature-local
|
||||
runtimes. Concretely:
|
||||
|
||||
1. **Promote shared capability Protocols** to the shared contracts module
|
||||
only when they have at least two production consumers. At decision
|
||||
time that meant `RpcCaller`, `LoopGuard`, `OperationScopeProvider`,
|
||||
and `AsyncWorkRuntime` in `_session_contracts.py`. Current code has
|
||||
demoted/deleted the single-consumer pieces: `_runtime/contracts.py`
|
||||
exports only `Kernel`, `RpcCaller`, and `LoopGuard`.
|
||||
|
||||
The symmetric demotion / sibling-fold rule lives in
|
||||
[ADR-0012 §Demotion / consolidation rule](0012-implementation-surface-convention.md#demotion-consolidation-rule):
|
||||
when a previously-shared capability drops back to a single consumer,
|
||||
or when two underscore-prefixed seam siblings become small enough
|
||||
and mutually-isolated, the ADR-0012 rule authorizes the inverse
|
||||
motion. The two rules read as a paired policy — ADR-0013 §1 governs
|
||||
promotion *up* into a shared Protocol; ADR-0012 §Demotion governs
|
||||
collapse *back down* into a feature-local Protocol or a folded
|
||||
seam file. Neither motion needs a fresh ADR; only changes to the
|
||||
criteria themselves do.
|
||||
|
||||
2. **Do not put account metadata on a broad runtime facade.** At decision
|
||||
time `AuthMetadata` and `Kernel` stayed as standalone Protocols in the
|
||||
shared contracts module. Current code keeps `Kernel` in
|
||||
`_runtime/contracts.py` as the typed transport surface and moves the
|
||||
single-consumer `AuthMetadata` Protocol local to `_source/upload.py`.
|
||||
|
||||
3. **Define feature-local runtime Protocols in their owning module** when
|
||||
a named composite earns its keep. At decision time these were:
|
||||
- `ChatRuntime` in `_chat.py` (composes `RpcCaller + LoopGuard`
|
||||
plus chat-only `transport_post(...)` and `next_reqid(...)`).
|
||||
- `ArtifactsRuntime` and `DrainHookRegistration` in `_artifacts.py`
|
||||
(composes `RpcCaller + AsyncWorkRuntime + DrainHookRegistration`).
|
||||
- `UploadRuntime` in `_source/upload.py` (historically `_source_upload.py`)
|
||||
(composes `RpcCaller +
|
||||
OperationScopeProvider + LoopGuard` plus `kernel` + `auth`
|
||||
constructor args).
|
||||
|
||||
Current code deleted those composites/adapters after ADR-0014 showed
|
||||
direct collaborator injection was clearer for their single consumers.
|
||||
|
||||
4. **Each feature constructor names its dependency by capability**, not
|
||||
by the broad `Session`:
|
||||
- Pure-RPC features (`NotebooksAPI`, `ResearchAPI`, `SettingsAPI`,
|
||||
`SharingAPI`) take `rpc: RpcCaller`.
|
||||
- Multi-capability features (`ChatAPI`, `ArtifactsAPI`,
|
||||
`SourceUploadPipeline`) take direct keyword-only collaborators
|
||||
(`rpc`, `transport`, `reqid`, `loop_guard`, `drain`, `lifecycle`,
|
||||
`kernel`, `auth`) rather than a broad or composite runtime object.
|
||||
|
||||
5. **Split `_mind_map.py`** into a private `NoteService` (new
|
||||
`_note_service.py`) and a `NoteBackedMindMapService` (mind-map
|
||||
adapter that stays in `_mind_map.py`). The pre-existing scaffolding
|
||||
the refactor relies on — `NotebookSourceLister`
|
||||
(`_notebook_metadata.py:19`), `NotebookSourceIdProvider`
|
||||
(`_notebook_metadata.py:26`), and the `_mind_map.py:55` service
|
||||
boundary — is reused. Module-level `_mind_map` wrappers are
|
||||
removed only after both collaborators are wired into `ArtifactsAPI`.
|
||||
|
||||
6. **Move saved-chat-from-`AskResult` ownership from `NotesAPI` to
|
||||
`ChatAPI`**: `ChatAPI.save_answer_as_note(notebook_id, ask_result, *,
|
||||
title: str | None = None) -> Note`. Define a `SaveChatAnswerCallback`
|
||||
type alias used by the deprecated `NotesAPI.create_from_chat`
|
||||
forwarder during the migration window. That forwarder was removed
|
||||
in v0.7.0; `ChatAPI.save_answer_as_note(...)` is the surviving API.
|
||||
|
||||
7. **No mixins for dependency expression.** Required capabilities are
|
||||
declared with `Protocol`s; extracted behavior is held by
|
||||
collaborators/services. This restates `docs/refactor-history.md` §Design
|
||||
Rule 6 and is binding for this refactor and future feature work.
|
||||
|
||||
8. **Underscore-prefix module privacy from
|
||||
[ADR-0012](0012-implementation-surface-convention.md) still applies.**
|
||||
This ADR does not change the public surface: `NotebookLMClient` and
|
||||
every `client.<feature>.<method>` reachable at v0.4.1 keeps its
|
||||
signature, defaults, and return type. Only private constructors and
|
||||
helper-module internals move.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted**
|
||||
|
||||
- Capability subsets are mypy-verified per feature. Adding a new
|
||||
feature picks the narrowest needed slice
|
||||
(`RpcCaller`, `LoopGuard`, `Kernel`, or a local Protocol) instead
|
||||
of opting into the entire `Session` surface.
|
||||
- Feature-local capabilities evolve without widening any shared union.
|
||||
If chat later needs another streaming primitive, the change is local
|
||||
to `_chat/` and the chat helpers, not to every feature that types
|
||||
against `RpcCaller`.
|
||||
- `_runtime/contracts.py` shrank after the migration arc deleted the
|
||||
broad `Session` Protocol and later demoted single-consumer shapes. It
|
||||
now contains only `Kernel`, `RpcCaller`, and `LoopGuard`.
|
||||
- The `auth/kernel/register_drain_hook` drift pattern is structurally
|
||||
prevented: `_runtime/contracts.py` accepts new shared Protocols only when a
|
||||
second consumer exists in the codebase at promotion time.
|
||||
- The note/mind-map service split lets Artifacts and Notes evolve their
|
||||
internal storage paths independently. The mind-map adapter can be
|
||||
rewritten without touching the generic note service.
|
||||
|
||||
**Unwanted**
|
||||
|
||||
- ≥36 direct feature-API test constructors must be updated alongside
|
||||
each migration step. Tests that instantiate `ChatAPI(session)`,
|
||||
`NotesAPI(session)`, `ArtifactsAPI(session)`, etc., must switch to
|
||||
the new keyword-only collaborator arguments. The migration steps in
|
||||
`docs/refactor-history.md` pair each feature retyping with its same-commit
|
||||
test fixture update so the build stays green.
|
||||
- The `_core.py` compatibility shim was removed in Phase 4 ([#889](https://github.com/teng-lin/notebooklm-py/pull/889)); see `tests/_guardrails/test_public_surface_manifest.py` (search for `Tier-10 PR-A re-export identity pins for ``notebooklm._core`` were deleted`) for the removal pin.
|
||||
- Two `RpcCaller` Protocols coexist briefly: the shared *object*
|
||||
protocol in `_runtime/contracts.py` (symbol `RpcCaller`, used by every
|
||||
feature API) and a pre-existing local *callable* protocol in
|
||||
`_source/upload.py` (historically `_source_upload.py`, symbol `RpcCallback`, used as the
|
||||
`register_file_source(rpc_call=...)` callback). They are structurally
|
||||
distinct (one is an object with an `rpc_call` method; the other is a
|
||||
callable). To avoid the name collision, the local callable protocol is
|
||||
**renamed** to `RpcCallback` in the same commit that introduces the
|
||||
shared `RpcCaller`. The local protocol is not deleted because the
|
||||
callback shape is a real seam the upload pipeline depends on.
|
||||
- This ADR exceeds the 250-line soft cap by user direction; the hard
|
||||
cap remains 500 lines. The expanded length is load-bearing: the
|
||||
decision shipping in this PR is the simultaneous adoption of three
|
||||
related changes (capability composition, note/mind-map split, and
|
||||
saved-chat ownership move) and they would not be coherent if split
|
||||
across multiple ADRs.
|
||||
|
||||
The incremental migration that realized this decision shipped across
|
||||
Phases 1–7 of the capability refactor arc and is now complete; this ADR
|
||||
records the architectural intent rather than the step sequence. See
|
||||
`docs/refactor-history.md` for the as-shipped module map and the
|
||||
narrative of how the cutover landed.
|
||||
|
||||
### Composed Runtime Consequences
|
||||
|
||||
- **C-X. `cookie_saver` / `cookie_rotator` late-binding seams** — introduced by PR [#879](https://github.com/teng-lin/notebooklm-py/pull/879) (`76b301d`). The cookie persistence hooks are now resolved through `_runtime/lifecycle.py` / `_runtime/init.py`, decoupling persistence logic from the core HTTP client transport and ensuring clean integration with the cookie keepalive loop.
|
||||
- **C-Y. Inline `__Secure-1PSIDTS` cold-start recovery** — introduced by PR [#872](https://github.com/teng-lin/notebooklm-py/pull/872) / issue #865 (`6d8b5f4`). Production utilizes `_recover_psidts_inline` in `_auth/psidts_recovery.py` to run a preflight healing check. If preconditions are met (e.g. not bypassing credentials in environment-driven auth, and utilizing a flock-based cross-process lock), the system proactively mints a valid `__Secure-1PSIDTS` cookie before initializing the session facade to prevent cold-start failures.
|
||||
- **C-Z. AST-based delegate-surface regression guard** *(historical)* — introduced by PR [#885](https://github.com/teng-lin/notebooklm-py/pull/885) (`f48d4b9`). It used AST analysis to pin simple delegate forwards on the now-deleted `Session` facade. Current guardrails instead prevent the deleted session/runtime-boundary surfaces from returning.
|
||||
- **C-AA. Drain-hook registration is owned by the transport drain tracker** — The standalone `DrainHookRegistration` Protocol from the Session/Kernel split was retired. *Current state (2026-06):* `DrainHookRegistration` is not a Protocol local to an `ArtifactsRuntime` (that composite no longer exists). The registration surface collapsed onto `TransportDrainTracker.register_drain_hook(...)` in `src/notebooklm/_transport_drain.py`. Feature code that owns long-running async work — artifact polling in `_artifact/polling.py` — registers its close-time hook there, and `ClientLifecycle.close` fires the registered hooks. See ADR-0014.
|
||||
- **C-AB. Narrow executor host Protocols after bridge retirement** *(historical)* — The session-shrink arc narrowed the former `RpcOwner` host Protocol after the legacy `Session` private-attribute shims were retired, dropping `_timeout`, `_refresh_callback`, `_refresh_retry_delay`, and `_http_client`. *Current state (2026-06):* `RpcOwner` was deleted entirely (session-decoupling Wave 4, #1068); `RpcExecutor` (`_rpc_executor.py`) now takes its `Kernel`, `RuntimeTransport`, `AuthRefreshCoordinator`, and `ClientMetrics` collaborators directly via keyword-only constructor parameters and keeps only a local `DecodeResponse` Protocol.
|
||||
- **C-AC. Retire the legacy transport Adapter** — The middleware chain terminal now calls `Kernel.post` through `MiddlewareChainHost._authed_post_chain_terminal -> RuntimeTransport.terminal`. Request construction lives in `_request_types.py`, transport exceptions and `Retry-After` parsing live in `_transport_errors.py`, and size-capped streaming lives in `_streaming_post.py`. This removed the last legacy host Protocol and the shallow Adapter seam.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
1. **Keep the broad `Session` contract and let it continue to grow.**
|
||||
Rejected. At ADR-write time the drift was already visible in
|
||||
`_session_contracts.py`'s broad `Session` Protocol: ADR-0010 specified
|
||||
five members, and the contract had grown to eight. Without an
|
||||
explicit promotion criterion (shared by ≥2 features), every future
|
||||
single-consumer capability is a candidate for promotion, and the
|
||||
contract is one PR away from nine members. The "narrow Session"
|
||||
intent of ADR-0010 is not recoverable by exhortation; it requires a
|
||||
structural rule.
|
||||
|
||||
2. **Per-sub-client narrow Protocols only, no feature-local runtimes**
|
||||
(the post-D2 replacement from the ADR-0002 lineage). Rejected.
|
||||
`ChatRuntime`'s `transport_post + next_reqid` slice cannot be
|
||||
modelled this way without either (a) widening a shared Protocol to
|
||||
include capabilities only chat uses, or (b) duplicating the
|
||||
`RpcCaller + LoopGuard` Protocol definitions across multiple
|
||||
feature modules. Composition via Protocol inheritance
|
||||
(`ChatRuntime(RpcCaller, LoopGuard, Protocol)`) gives the same
|
||||
narrow-typing benefit without either downside.
|
||||
|
||||
3. **Promote `transport_post` and `next_reqid` globally to shared
|
||||
Protocols.** Rejected. There is no second consumer today. Promoting
|
||||
on speculation is exactly the drift pattern this ADR fights:
|
||||
`auth`/`kernel`/`register_drain_hook` were promoted to the broad
|
||||
`Session` for "convenience" without a second-consumer trigger, and
|
||||
the result was the eight-member contract enumerated in the Context
|
||||
section above. The promotion criterion in Decision §1
|
||||
(shared by ≥2 features) exists precisely to block this path. If a
|
||||
future feature genuinely needs chat's transport slice, the
|
||||
capability is promoted at that point with the second consumer as
|
||||
evidence.
|
||||
|
||||
4. **Split into multiple ADRs (capability composition + note/mind-map
|
||||
split + saved-chat ownership move).** Rejected per user direction.
|
||||
The three changes are not independent: the note/mind-map split is a
|
||||
prerequisite for `ArtifactsAPI` taking `mind_maps` and
|
||||
`note_service` collaborators (Decision §5), and the saved-chat
|
||||
ownership move depends on `ChatAPI` already taking a `ChatRuntime`
|
||||
(Decision §6). Splitting the ADR would either force a fragile
|
||||
inter-ADR ordering or hide the unified design intent behind three
|
||||
smaller records that each look like "just a refactor". One ADR
|
||||
keeps the decision narrative intact and explicit about the three
|
||||
changes shipping together.
|
||||
@@ -0,0 +1,538 @@
|
||||
# ADR-0014: Feature-local runtime adapters as Protocol satisfiers
|
||||
|
||||
> **Current state (2026-06-11).** The normative body below describes the state
|
||||
> *at decision time*, when a concrete `Session` facade class and module still
|
||||
> existed. **`Session` and `_session.py` have since been deleted** (see
|
||||
> [Revision history](#revision-history) → "2026-05-28 — Session elimination"),
|
||||
> and the former `_session_*` / `_runtime_*` collaborators now live under the
|
||||
> `_runtime/` package (for example `_runtime/init.py`,
|
||||
> `_runtime/transport.py`; the `SessionTransport` / `SessionCollaborators`
|
||||
> classes are now `RuntimeTransport` / `RuntimeCollaborators`). The
|
||||
> feature-local composite Protocols and adapter dataclasses discussed below
|
||||
> were also retired when direct keyword-only collaborator injection proved
|
||||
> clearer for their single consumers. Treat in-body references to `Session`,
|
||||
> `_session.py`, `_session_*`, and exact `client.py:NNN` line numbers as
|
||||
> historical; the
|
||||
> live runtime shape is documented in
|
||||
> [`docs/architecture.md`](../architecture.md).
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (#1082; Stage-B issue #1084; MiddlewareChainHost issue #1085).
|
||||
Rule 3 Stage B closed by the post-refactoring plan 2026-05-27 Stage B1
|
||||
(#1086 / #1089 / #1091). Rule 4 deepening closed by the same plan's
|
||||
Stage B2 (#1090 / #1092 / this PR) — the chain ownership carve-out is
|
||||
recorded under [Revision history](#revision-history) below.
|
||||
|
||||
## Context
|
||||
|
||||
[ADR-0013](./0013-composable-session-capabilities.md) introduced narrow capability Protocols
|
||||
(`RpcCaller`, `LoopGuard`, `OperationScopeProvider`, `AuthMetadata`, `Kernel` in
|
||||
the then-current `_session_contracts.py`) plus feature-local
|
||||
composite runtime Protocols (`ChatRuntime` in `_chat.py`, `ArtifactsRuntime` in
|
||||
`_artifacts.py`, `UploadRuntime` in `_source_upload.py`). The goal was to decouple
|
||||
feature APIs from a concrete `Session` god-object.
|
||||
|
||||
At compile time, the goal was achieved. Every feature API type-checks against the
|
||||
narrowest Protocol it needs, mypy verifies the satisfaction, and the
|
||||
the contracts module docstring enforces the "≥2 consumers ⇒ shared
|
||||
Protocol; otherwise feature-local" promotion rule. Current shared
|
||||
contracts live in `_runtime/contracts.py` and are only `Kernel`,
|
||||
`RpcCaller`, and `LoopGuard`; `AuthMetadata` and `OperationScopeProvider`
|
||||
are local to their only consumers.
|
||||
|
||||
At runtime, the goal was not achieved. `NotebookLMClient.__init__`
|
||||
([`client.py:305-342`](../../src/notebooklm/client.py)) passes `self._session` (a
|
||||
`Session` instance) to every feature API. `Session` was kept as the universal
|
||||
satisfier of every Protocol. This produces four observable consequences:
|
||||
|
||||
1. **`Session` must satisfy the union of every feature Protocol.** Adding a method
|
||||
to `ChatRuntime` requires `Session` to expose (or forward) that method.
|
||||
`Session`'s method count grows with the feature count by construction. The
|
||||
current count is ~33 methods on a 779-line class; ~24 of those are
|
||||
one-line forwards to a held collaborator.
|
||||
|
||||
2. **Forwards are structurally required, not accidental.** `Session.transport_post`
|
||||
exists because `ChatRuntime` requires it — `Session` must forward to
|
||||
`SessionTransport`. Deleting the forward is not possible while `Session`
|
||||
satisfies `ChatRuntime`. Earlier "shrink the facade" attempts (the
|
||||
in-flight rpc-dispatch refactor; the Stage 2 work in the architecture
|
||||
fix-plan that produced PRs #1049/#1059/#1061) consistently hit this wall.
|
||||
|
||||
3. **Tests monkeypatch `Session`, not the collaborator.** At runtime, the method
|
||||
body lives on `Session`. Tests that need to fake `transport_post` patch
|
||||
`Session.transport_post`, not `SessionTransport.perform_authed_post`. The
|
||||
[ADR-0007](./0007-test-monkeypatch-policy.md) forbidden-monkeypatch
|
||||
allowlist (~30 file-level entries today) is the visible gravity well this
|
||||
creates — every entry pins a Session-shaped surface that ADR-0013's narrow
|
||||
Protocols _should_ have eliminated.
|
||||
|
||||
4. **`RpcOwner` Protocol carries underscore-prefixed `Session` internals.**
|
||||
`RpcExecutor` declares an `RpcOwner` dependency
|
||||
([`_rpc_executor.py:59-78`](../../src/notebooklm/_rpc_executor.py)) listing
|
||||
`_kernel`, `_perform_authed_post`, `_await_refresh`, `_increment_metrics`.
|
||||
This is not "narrow contract"; it is "private surface of `Session`, structurally
|
||||
typed". The leakage persists because `RpcExecutor` receives a `Session`-shaped
|
||||
owner at construction.
|
||||
|
||||
The architectural pressure is real and ongoing. Every feature added between v0.5.0
|
||||
and today has either grown `Session` or required a new compatibility forward.
|
||||
ADR-0013 framed the _interface_ model correctly; what was missing is the matching
|
||||
_implementation_ model.
|
||||
|
||||
## Decision
|
||||
|
||||
Capability Protocols remain conceptually as defined by ADR-0013, but their
|
||||
homes changed after the migration. Current shared contracts live in
|
||||
`_runtime/contracts.py`, and single-consumer capabilities live beside their
|
||||
owners. **This ADR does not change the public API.** Six
|
||||
implementation rules change how those interfaces are satisfied at runtime.
|
||||
|
||||
### Rule 1 — Single-collaborator Protocols are satisfied directly (after method push-down)
|
||||
|
||||
| Protocol | Satisfier (post-migration) | Migration prerequisite |
|
||||
| -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `RpcCaller` | `RpcExecutor` directly | none — already structurally satisfies (TYPE_CHECKING assertion at [`_rpc_executor.py:463`](../../src/notebooklm/_rpc_executor.py)) |
|
||||
| `LoopGuard` | `ClientLifecycle` directly | **push down `assert_bound_loop()`** — currently lives on `Session.assert_bound_loop` (`_session.py:486`), which calls `_loop_affinity.assert_bound_loop(self.bound_loop)`. `ClientLifecycle` already owns `get_bound_loop` (`_session_lifecycle.py:271`); the push-down adds a trivial `assert_bound_loop()` method that calls the free function with `self.get_bound_loop()`. |
|
||||
| `OperationScopeProvider` | `TransportDrainTracker` directly | **push down `operation_scope(label)`** — currently lives on `Session.operation_scope` (`_session.py:495`) as an async context manager wrapping `begin_transport_post` / `finish_transport_post` (both already on `TransportDrainTracker` at [`_transport_drain.py:139,196`](../../src/notebooklm/_transport_drain.py)). The push-down moves the contextmanager wrapper to the tracker. |
|
||||
| `DrainHookRegistration` (feature-local in `_artifacts.py`) | `TransportDrainTracker` directly | **push down `register_drain_hook(name, hook)` + the underlying `_drain_hooks` storage** — currently lives on `Session.register_drain_hook` (`_session.py:421`). The push-down moves both the method and the storage onto the tracker. |
|
||||
| `AsyncWorkRuntime` (composes `LoopGuard` + `OperationScopeProvider`) | satisfied **transitively** by `ArtifactsRuntimeAdapter` / `UploadRuntimeAdapter` (Rule 2) | depends on the push-downs above. No dedicated `_AsyncWorkAdapter` — per Rule 2, trivial composites do not get adapter middlemen. |
|
||||
| `AuthMetadata` | `AuthTokens` (`client._auth`) directly | `SourceUploadPipeline` receives the client-owned `AuthTokens` via `auth=client._auth`; `AuthTokens` structurally provides the `authuser` and `account_email` members required by `_source/upload.py::AuthMetadata`. |
|
||||
| `Kernel` (Protocol) | the concrete `Kernel` class | none — unchanged |
|
||||
|
||||
**Why the push-downs are part of this ADR's mandate, not pre-existing.** The
|
||||
Protocol satisfiers were synthesised on `Session` because `Session` was the
|
||||
universal satisfier. Now that the runtime contract is "the collaborator
|
||||
satisfies its own Protocol", the methods must live where the Protocol points.
|
||||
The push-downs are mechanical (the underlying primitives already exist on the
|
||||
collaborators) but they are non-trivial enough to be sequenced as a dedicated
|
||||
migration wave (Wave 0.5 — push Protocol-satisfying methods down to
|
||||
collaborators) ahead of the adapter introductions.
|
||||
|
||||
### Rule 2 — Composite Protocols are satisfied by a feature-local adapter when the adapter earns its keep
|
||||
|
||||
**Adapter threshold (intent-based).** Introduce a frozen-dataclass adapter when
|
||||
**at least one** of the following holds:
|
||||
|
||||
- a downstream module _intentionally_ consumes the composite Protocol as a
|
||||
single dependency (e.g. an HTTP-level helper that takes the whole runtime),
|
||||
**or**
|
||||
- delegation changes the call shape (the adapter method does more than forward
|
||||
args 1:1 to a single underlying collaborator), **or**
|
||||
- multiple consumers share the same composite and the share-cost amortises.
|
||||
|
||||
If none of these hold — the composite has a single consumer, all delegates are
|
||||
1:1, and no other module takes it as a parameter — the consumer takes the
|
||||
underlying collaborators directly via constructor injection. No adapter
|
||||
middleman.
|
||||
|
||||
The earlier numeric heuristic ("≥3 capabilities OR ≥1 non-trivial delegate")
|
||||
was a proxy for the intent test above. It is replaced because counting
|
||||
capabilities does not capture _why_ you would want a named runtime satisfier:
|
||||
either some consumer demands it, or the call shape adapts. Counting alone
|
||||
incentivises adapters where they bring no value.
|
||||
|
||||
When an adapter is used, the frozen dataclass holds the collaborators a
|
||||
composite Protocol requires and exposes the Protocol methods as delegates.
|
||||
Adapters live in the same module as the Protocol they satisfy.
|
||||
|
||||
**Corollary — dead Protocols.** If a composite Protocol is no longer consumed
|
||||
(no downstream module takes it as a parameter; its sole consumer was migrated
|
||||
to direct injection), delete the Protocol along with the migration. Protocols
|
||||
that exist only as type-hint documentation drift out of sync with the code.
|
||||
|
||||
```python
|
||||
# in _artifacts.py, next to ArtifactsRuntime
|
||||
@dataclass(frozen=True)
|
||||
class ArtifactsRuntimeAdapter:
|
||||
"""Concrete satisfier of :class:`ArtifactsRuntime` per ADR-0014.
|
||||
|
||||
Earns its keep under Rule 2 because:
|
||||
- composite has 3 capabilities (RpcCaller + AsyncWorkRuntime + DrainHookRegistration),
|
||||
- ArtifactsAPI takes the whole composite as a single dependency (consumer-demand),
|
||||
- register_drain_hook is a meaningful named affordance worth exposing as one method.
|
||||
"""
|
||||
|
||||
rpc: RpcCaller
|
||||
drain: TransportDrainTracker # satisfies OperationScopeProvider + DrainHookRegistration after Wave 0.5
|
||||
lifecycle: ClientLifecycle # satisfies LoopGuard after Wave 0.5
|
||||
|
||||
async def rpc_call(self, *args: Any, **kwargs: Any) -> Any:
|
||||
return await self.rpc.rpc_call(*args, **kwargs)
|
||||
|
||||
def operation_scope(self, label: str) -> AbstractAsyncContextManager[None]:
|
||||
return self.drain.operation_scope(label)
|
||||
|
||||
def register_drain_hook(self, name: str, hook: Callable[[], Awaitable[None]]) -> None:
|
||||
self.drain.register_drain_hook(name, hook)
|
||||
|
||||
def assert_bound_loop(self) -> None:
|
||||
self.lifecycle.assert_bound_loop()
|
||||
```
|
||||
|
||||
`UploadRuntimeAdapter` follows the same pattern in `_source_upload.py`
|
||||
(`SourceUploadPipeline` already takes `Kernel` and `AuthMetadata` as separate
|
||||
parameters, so its adapter covers only the composite Protocol part).
|
||||
|
||||
`ChatRuntime` does **not** get an adapter — the Rule 2 Corollary applies. Once
|
||||
`_chat_transport.chat_aware_authed_post` is refactored to take `SessionTransport`
|
||||
directly (Wave 4.1 Step 0), `ChatRuntime` has no remaining consumer and is
|
||||
deleted. `ChatAPI` takes the four underlying collaborators (`RpcExecutor`,
|
||||
`SessionTransport`, `ReqidCounter`, `ClientLifecycle`) as keyword-only
|
||||
constructor parameters.
|
||||
|
||||
### Rule 3 — `NotebookLMClient.__init__` is the composition root
|
||||
|
||||
`NotebookLMClient.__init__` wires each feature with its satisfier — a
|
||||
collaborator for single-Protocol features, an adapter for composite ones.
|
||||
Construction reads as an explicit wiring diagram.
|
||||
|
||||
The migration ships in two stages so the structural change does not balloon:
|
||||
|
||||
- **Stage A (this plan):** `Session` exposes the collaborator bundle as a
|
||||
single typed attribute (`Session.collaborators -> SessionCollaborators`).
|
||||
`NotebookLMClient.__init__` reads `self._session.collaborators.<x>` for
|
||||
feature wiring. `Session` continues to construct the bundle internally via
|
||||
`_session_init.build_collaborators`.
|
||||
- **Stage B (Wave 7 follow-up, separate change):** ownership of
|
||||
`build_collaborators` moves to `NotebookLMClient.__init__`. `Session` takes
|
||||
the bundle as a constructor argument. This is the strategic end-state but is
|
||||
a wider blast radius (every `Session(...)` test-construction site updates)
|
||||
and is deferred so the in-flight ADR-0014 migration stays bounded.
|
||||
|
||||
Stage A is sufficient to discharge the runtime decoupling: features stop
|
||||
receiving `Session` (they receive the collaborator or adapter directly), and
|
||||
the only `Session` reference at the composition root is `self._session.collaborators`,
|
||||
which is one attribute access — not a discoverability hub of method names.
|
||||
|
||||
### Rule 4 — `Session` retains only orchestration plus the documented middleware-chain seams
|
||||
|
||||
After migration, `Session` owns:
|
||||
|
||||
- Construction delegation (`__init__` → `_session_init.build_collaborators`).
|
||||
- Lifecycle (`open`, `close`, keepalive task, `is_open`, drain-on-close).
|
||||
- The collaborator graph held as attributes.
|
||||
- The few methods downstream tests still pin via AST-guard
|
||||
(`tests/_guardrails/test_public_surface_manifest.py`,
|
||||
`tests/unit/test_concurrency_refresh_race.py`).
|
||||
- **Live middleware-chain seams that legitimately route through `Session`.**
|
||||
**Chain-ownership carve-out (post-refactoring plan 2026-05-27 Stage B2):**
|
||||
the storage backing the chain's tunables — `_authed_post_chain_terminal`,
|
||||
`_authed_post_chain`, `_rate_limit_max_retries`,
|
||||
`_server_error_max_retries`, `_refresh_retry_delay` — moved off `Session`
|
||||
onto `MiddlewareChainHost`
|
||||
([`_middleware_chain_host.py`](../../src/notebooklm/_middleware/chain_host.py))
|
||||
in Stage B2 PR 1 (#1090). PR 2 (#1092) then split
|
||||
`wire_middleware_chain` / `build_session_transport` to take
|
||||
`chain_host: MiddlewareChainHost` directly, so the live chain reads
|
||||
the host and the Session-side names are exclusively writable
|
||||
`@property` descriptors that forward to the host (test-seam only;
|
||||
no longer the load-bearing dereference on the hot path). The
|
||||
long-standing fixture-rebind contract is preserved end-to-end — a test
|
||||
that writes `core._<attr> = ...` still steers the live chain because
|
||||
the descriptor's setter writes through to `chain_host._<attr>`. The
|
||||
per-attr contract is now historical; current code binds the host directly
|
||||
from `ClientComposed`. `Session.assert_bound_loop` remains captured via lambda by
|
||||
`build_session_transport`'s `bound_loop_check` (not migrated to the
|
||||
host — it forwards to `ClientLifecycle` per Rule 1).
|
||||
|
||||
`Session` stops being passed _to_ feature APIs. It stops satisfying capability
|
||||
Protocols intended for feature consumption. The compatibility forwards on
|
||||
`Session` (drain/metrics/kernel/authuser/save_cookies forwards that exist only
|
||||
because features used to reach through `Session`) are removed as Wave 5
|
||||
deletes them.
|
||||
|
||||
**Explicit retention list** — surfaces that remain on `Session` post-migration:
|
||||
|
||||
- `Session._chain_host` — instance reference to the `MiddlewareChainHost`
|
||||
that owns the live middleware chain. The host owns
|
||||
`_authed_post_chain_terminal` (chain leaf), `_authed_post_chain`
|
||||
(installed chain slot), the three retry-budget tunables
|
||||
(`_rate_limit_max_retries`, `_server_error_max_retries`,
|
||||
`_refresh_retry_delay`), and the dynamic `await_refresh` delegate.
|
||||
`wire_middleware_chain` and `build_session_transport` take
|
||||
`chain_host: MiddlewareChainHost` directly; the chain's provider
|
||||
lambdas and the transport's `chain_provider` closure read the host
|
||||
attributes live. Tests rebind through `core._chain_host._<attr>`.
|
||||
The transitional Session-side writable `@property` descriptors and
|
||||
the `Session._await_refresh` delegate that briefly forwarded reads
|
||||
/ writes to the host during the chain-ownership carve-out have been
|
||||
removed; the host is now the sole owner with no Session-side aliases.
|
||||
- `Session.update_auth_tokens` — AST-guarded by `test_concurrency_refresh_race.py`.
|
||||
- `Session.open` / `close` / `is_open` / `_keepalive_loop` — lifecycle.
|
||||
- ~~`Session.collaborators`, `Session.session_transport`, `Session.rpc_executor`~~
|
||||
— three typed accessors per Rule 3 Stage A. **Deleted under Rule 3 Stage B
|
||||
by Stage B1 PR 2 of the post-refactoring plan 2026-05-27 (#1089 commit
|
||||
`313bbef1`)** — `compose_session_internals()` became the composition root
|
||||
and later moved to `_session_init.py` during Session-elimination Phase 1;
|
||||
the three accessor properties and the `_get_rpc_executor` lazy factory all
|
||||
collapsed to direct reads on the returned `ComposedSession`. The previous
|
||||
Task 6.3 AST lint protecting these accessors no longer has live callers to
|
||||
protect.
|
||||
|
||||
Everything else on `Session` after Wave 5 should be either listed here or
|
||||
deleted.
|
||||
|
||||
### Rule 5 — Collaborators take their direct dependencies
|
||||
|
||||
Any collaborator that currently dereferences `self._owner.X` is migrated to
|
||||
take its actual dependencies directly. `RpcExecutor` is the canonical case:
|
||||
|
||||
```python
|
||||
# before
|
||||
class RpcExecutor:
|
||||
def __init__(self, owner: RpcOwner, *, decode_response, is_auth_error, sleep, ...):
|
||||
self._owner = owner
|
||||
|
||||
async def _execute_once(self, ...):
|
||||
self._owner._kernel.get_http_client() # pre-open guard
|
||||
self._owner._increment_metrics(...)
|
||||
response = await self._owner._perform_authed_post(...)
|
||||
await self._owner._await_refresh()
|
||||
|
||||
# after
|
||||
class RpcExecutor:
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
kernel: Kernel,
|
||||
transport: SessionTransport,
|
||||
auth_refresh: AuthRefreshCoordinator,
|
||||
metrics: ClientMetrics,
|
||||
decode_response, is_auth_error, sleep, ...
|
||||
):
|
||||
self._kernel = kernel
|
||||
self._transport = transport
|
||||
self._auth_refresh = auth_refresh
|
||||
self._metrics = metrics
|
||||
|
||||
async def _execute_once(self, ...):
|
||||
self._kernel.get_http_client() # same guard, direct
|
||||
self._metrics.increment(...)
|
||||
response = await self._transport.perform_authed_post(...)
|
||||
await self._auth_refresh.await_refresh()
|
||||
```
|
||||
|
||||
The `RpcOwner` Protocol disappears. Same exercise for any other collaborator
|
||||
holding an `_owner` reference.
|
||||
|
||||
### Rule 6 — Adapters live next to their feature
|
||||
|
||||
`ArtifactsRuntimeAdapter` lives in `_artifacts.py`. `UploadRuntimeAdapter` lives
|
||||
in `_source_upload.py`. They are concrete implementations of feature-local
|
||||
composite Protocols.
|
||||
|
||||
The Chat feature gets no adapter (Rule 2 Corollary): once
|
||||
`_chat_transport.chat_aware_authed_post` is refactored to take `SessionTransport`
|
||||
directly, `ChatRuntime` has no remaining consumer and is deleted. `ChatAPI`
|
||||
takes the underlying collaborators as keyword-only constructor parameters
|
||||
instead.
|
||||
|
||||
The ADR-0013 promotion rule (≥2 consumers ⇒ shared Protocol in
|
||||
`_runtime/contracts.py`) is unchanged. Adapters are _not_ promoted to the
|
||||
shared contracts module; it stays interface-only.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Migration outcome:** Migration completed in PRs #1064–#1082; ADR-0007 Session-shaped allowlist entries drained; later Session-elimination work moved the remaining lifecycle/public-surface duties onto `NotebookLMClient` and client-owned collaborators. The two Wave-7 follow-ups (#1084 Stage B and #1085 MiddlewareChainHost) were closed by the post-refactoring plan 2026-05-27 — Stage B1 (#1086 / #1089 / #1091) and Stage B2 (#1090 / #1092 / this PR) respectively. See [Revision history](#revision-history) for the chain-ownership carve-out introduced under Stage B2 and the 2026-05-28 elimination note.
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- The deleted `Session` method count stopped growing with feature count.
|
||||
New features receive direct collaborators, or a feature-local adapter only
|
||||
when Rule 2's "earns its keep" test is met.
|
||||
- `RpcOwner` Protocol disappears entirely. No more underscore-prefixed
|
||||
Session-internal members in a "narrow" contract.
|
||||
- Tests fake the adapter or single collaborator, not `Session`. The ADR-0007
|
||||
forbidden-monkeypatch allowlist becomes drainable — the surface tests pinned
|
||||
(Session method bodies) is gone.
|
||||
- Independent feature testability. A test for `ChatAPI` constructs a narrow
|
||||
set of fake collaborators (just a fake `RpcExecutor`, depending on what the
|
||||
test exercises). It does not need to know `Session` exists.
|
||||
- Construction is explicit. `NotebookLMClient.__init__` reads as a wiring
|
||||
diagram. Replaces "Session has every method" with "every feature gets exactly
|
||||
what it asked for".
|
||||
- Closes ADR-0013's runtime story. ADR-0013 framed the interface model; this ADR
|
||||
completes the implementation model.
|
||||
|
||||
**Unwanted:**
|
||||
|
||||
- Each new feature requires a new adapter (5-10 lines). Small ongoing cost. The
|
||||
cost is local to the feature module and visible at construction time — preferable
|
||||
to invisible growth of `Session`.
|
||||
- Wider client composition wiring. Mitigated by
|
||||
`_client_assembly.py::_assemble_client(...)` and `_runtime/init.py`
|
||||
centralizing collaborator construction.
|
||||
- Migration churn for existing tests. Tests that constructed a `Session` and then
|
||||
patched a method must migrate to fake-adapter construction. The ADR-0007
|
||||
program already pays for this migration; this ADR aligns the destination.
|
||||
- Adapter method bodies are formally one-line forwards. They could be
|
||||
auto-generated. We do not — explicit method bodies keep the Protocol contract
|
||||
visible at the satisfier and let mypy catch shape mismatches at the adapter.
|
||||
|
||||
**Neutral:**
|
||||
|
||||
- `Session` may eventually be renamed (e.g. `_SessionLifecycle`, `_ClientCore`)
|
||||
once it owns only lifecycle and the collaborator graph. Out of scope for this
|
||||
ADR; defer until after the migration completes and the right name is obvious.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Auto-generate forwards via `__getattr__`.** Rejected. Hides the coupling
|
||||
instead of solving it. Tests still monkeypatch the auto-generated surface;
|
||||
`Session`'s effective surface stays wide.
|
||||
|
||||
- **Dataclass-Session with method bodies on collaborators.** Equivalent end
|
||||
state, framed differently. We picked the feature-local-adapter framing
|
||||
because it co-locates the adapter with the feature that consumes it
|
||||
(single-file readability) and lets each feature evolve its runtime
|
||||
independently. A renamed-Session-as-dataclass-bag is still an option later,
|
||||
but the adapter pattern is the primitive that unlocks it.
|
||||
|
||||
- **Per-feature god-objects (Session-per-feature).** Each feature gets a narrow
|
||||
facade that owns its method bodies. Rejected — moves the god-object problem
|
||||
from one location to N, and doesn't share collaborator instances across
|
||||
features cleanly.
|
||||
|
||||
- **Keep `Session` as universal satisfier; drain the ADR-0007 allowlist by
|
||||
case-by-case exception.** Rejected. The allowlist exists because `Session`'s
|
||||
method surface is the test gravity well. Draining the allowlist while the
|
||||
gravity persists is treating the symptom; it grows back as new features are
|
||||
added.
|
||||
|
||||
- **Stop the decomposition; embrace `Session` as god-object.** Rejected. Gives
|
||||
up ADR-0013's gains. The maintenance cost of `Session` has been measurable
|
||||
across the multi-phase refactor program.
|
||||
|
||||
## Migration
|
||||
|
||||
The migration ran as a staged series of per-wave PRs, recorded in the
|
||||
[Revision history](#revision-history) below; the standalone implementation
|
||||
plan that scoped those waves has been retired now that the work is complete.
|
||||
The post-refactoring plan 2026-05-27 (Stage B1 + Stage B2) closed the two
|
||||
deferred follow-ups — see [Revision history](#revision-history).
|
||||
|
||||
## Revision history
|
||||
|
||||
### 2026-05-27 — Rule 3 Stage B closure (post-refactoring plan 2026-05-27 Stage B1, #1086 / #1089 / #1091)
|
||||
|
||||
Issue #1084 (deferred Rule 3 Stage B) closed. `compose_session_internals()`
|
||||
became the composition root and later became
|
||||
`_runtime/init.py::compose_client_internals`:
|
||||
`Session.__init__` was
|
||||
narrowed to `(*, collaborators, config, auth)` and the Stage A accessor
|
||||
properties (`Session.collaborators`, `Session.session_transport`,
|
||||
`Session.rpc_executor`) plus the `_get_rpc_executor` lazy factory were
|
||||
deleted. Feature wiring now reads `composed.collaborators` /
|
||||
`composed.transport` / `composed.executor` from the `ComposedSession`
|
||||
returned by the helper. The deletion is recorded in the **Deleted**
|
||||
section of this ADR's revision history.
|
||||
|
||||
### 2026-05-27 — Rule 4 chain-ownership carve-out (post-refactoring plan 2026-05-27 Stage B2, #1090 / #1092)
|
||||
|
||||
Issue #1085 (deferred `MiddlewareChainHost` extraction) closed.
|
||||
|
||||
- **#1090** introduced
|
||||
[`_middleware_chain_host.py`](../../src/notebooklm/_middleware/chain_host.py).
|
||||
The chain's tunable storage (`_authed_post_chain_terminal`,
|
||||
`_authed_post_chain`, `_rate_limit_max_retries`,
|
||||
`_server_error_max_retries`, `_refresh_retry_delay`) moved from
|
||||
`Session` instance attributes onto `MiddlewareChainHost` fields.
|
||||
Session retained the five names as writable `@property` descriptors
|
||||
that forward to the host so the long-standing fixture-rebind seam
|
||||
(`core._<attr> = ...` writing through to the live chain) survived
|
||||
the move. `_await_refresh` was routed through
|
||||
`MiddlewareChainHost.await_refresh` (dynamic delegation to
|
||||
`host._auth_refresh.await_refresh()`) for the same reason.
|
||||
- **#1092** split
|
||||
`_runtime/init.py::wire_middleware_chain` / `build_runtime_transport`
|
||||
to take `chain_host: MiddlewareChainHost` directly. The chain's
|
||||
provider lambdas (`chain_provider`,
|
||||
`rate_limit_max_retries_provider`,
|
||||
`server_error_max_retries_provider`, `refresh_retry_delay_provider`)
|
||||
and the transport's `chain_provider` closure now dereference
|
||||
`chain_host.<attr>` directly on every attempt, instead of routing
|
||||
through the Session descriptors.
|
||||
|
||||
### 2026-05-27 — Carve-out close-out (delete Session-side chain descriptors)
|
||||
|
||||
The Session-side writable `@property` descriptors
|
||||
(`_authed_post_chain_terminal`, `_authed_post_chain`,
|
||||
`_rate_limit_max_retries`, `_server_error_max_retries`,
|
||||
`_refresh_retry_delay`) and the `_await_refresh` delegate that briefly
|
||||
bridged tests to `MiddlewareChainHost` during the carve-out have been
|
||||
deleted. The chain host is the sole owner; the composition root
|
||||
addresses it directly (`chain_host._authed_post_chain =
|
||||
wired.authed_post_chain`, `authed_post_chain_terminal=chain_host._authed_post_chain_terminal`)
|
||||
and tests rebind through `core._chain_host._<attr>` /
|
||||
`core._chain_host.await_refresh()`. The Session-side retention list
|
||||
keeps only the `_chain_host` reference (so feature code and tests can
|
||||
reach the host); the descriptors and the `_await_refresh` delegate
|
||||
are recorded in the historical deletion notes below.
|
||||
|
||||
### 2026-05-27 — Rule 2 adapter retirement (runtime-adapter decision)
|
||||
|
||||
The Rule 2 example dataclasses `ArtifactsRuntimeAdapter` and
|
||||
`UploadRuntimeAdapter` introduced for the artifact and upload features
|
||||
were retired. Each adapter only hid three stable collaborators
|
||||
(`RpcCaller` + `TransportDrainTracker` + `ClientLifecycle`) and had
|
||||
exactly one production satisfier, so they sat at the bottom of Rule
|
||||
2's keep-vs-delete spectrum. The feature constructors now take their
|
||||
three runtime collaborators (`rpc` + `drain` + `lifecycle`) as
|
||||
keyword-only arguments directly — mirroring the post-ADR-0014 `ChatAPI`
|
||||
pattern. The feature-local composite Protocols (`ArtifactsRuntime`,
|
||||
`UploadRuntime`) and the local `DrainHookRegistration` Protocol were
|
||||
deleted with their adapters; the mypy structural-satisfier guards near
|
||||
the adapter definitions are no longer needed because the constructors
|
||||
type each slot against its narrow shared Protocol directly. The
|
||||
post-migration table and example code earlier in this ADR document the
|
||||
historical Rule 2 satisfier pattern; this revision-history note is the
|
||||
authoritative current-state pointer.
|
||||
|
||||
### 2026-05-28 — Host-protocol removal (plan `host-protocol-removal`)
|
||||
|
||||
Lifecycle / auth refresh no longer consume Session-shaped host
|
||||
protocols (host-protocol-removal plan, Waves 1-3, PRs #1131-#1134).
|
||||
The `_LifecycleHost` and `RefreshAuthCore` Protocols and the
|
||||
`typing.cast(_LifecycleHost, core)` site in `_auth/session.py` were
|
||||
deleted in Wave 2 (PR #1133); `refresh_auth_session` now takes the
|
||||
five concrete collaborators (`auth`, `kernel`, `auth_coord`,
|
||||
`lifecycle`, `cookie_persistence`) as keyword-only arguments and
|
||||
`ClientLifecycle.save_cookies` takes the `CookiePersistence`
|
||||
collaborator directly. Wave 3 (PR #1134) deleted the Session-level
|
||||
auth/lifecycle forwards (`Session.lifecycle`, `Session.update_auth_tokens`,
|
||||
`Session.update_auth_headers`) the retired Protocols had backed;
|
||||
`NotebookLMClient.auth` now reads `self._auth` directly (set in
|
||||
`__init__`) and `SourceUploadPipeline(auth=self._auth)` is wired the
|
||||
same way. Wave 4 (PR for this revision) added regression lints under
|
||||
`tests/_guardrails/test_session_runtime_boundaries.py` plus extensions to
|
||||
`test_client_composition.py` and `test_no_session.py` so neither host
|
||||
Protocol nor the deleted Session forwards can quietly come back.
|
||||
The auth-refresh path is now fully explicit-collaborator-driven and
|
||||
ADR-0014 Rule 3 holds end-to-end on the refresh code path as it
|
||||
already did on the feature constructors.
|
||||
|
||||
### 2026-05-28 — Session elimination (plan `session-elimination-plan`)
|
||||
|
||||
`NotebookLMClient` now owns the final runtime graph directly: `ClientComposed`,
|
||||
the `SessionCollaborators` bundle, the `RpcExecutor`, and the public feature
|
||||
APIs. The concrete `Session` class and its module were deleted, along with the
|
||||
session-method retention document and helper factory. Lifecycle entry points
|
||||
(`__aenter__`, `__aexit__`, `close`, `drain`, and `is_connected`) call
|
||||
`ClientLifecycle` and `TransportDrainTracker` directly. Static lints now enforce
|
||||
that the deleted module, deleted helper names, deleted client attribute, and
|
||||
`ClientComposed.collaborators` alias cannot return.
|
||||
|
||||
## Related decisions
|
||||
|
||||
- Builds on [ADR-0013](./0013-composable-session-capabilities.md) (capability
|
||||
Protocol pattern). This ADR is the runtime-side completion of ADR-0013's
|
||||
interface-side decoupling.
|
||||
- Enables completion of the [ADR-0007](./0007-test-monkeypatch-policy.md)
|
||||
allowlist drain by removing the surface those entries currently pin.
|
||||
- Closes the deferred goal in [ADR-0003](./0003-auth-facade-write-through.md)
|
||||
by example — `auth.py` follows the same delegate-to-private-module pattern
|
||||
Rule 1 applies to collaborators.
|
||||
- Supersedes the former "Session as facade" and lifecycle-root framing in
|
||||
[`docs/architecture.md`](../architecture.md); the current architecture is
|
||||
"NotebookLMClient as composition root".
|
||||
@@ -0,0 +1,313 @@
|
||||
# ADR-0015: Typed JSON error envelope covers post-parse `ClickException` failures
|
||||
|
||||
## Status
|
||||
|
||||
Accepted. **Amended 2026-06-02** — see the amendment note below; it supersedes
|
||||
the original "no JSON emitted at parse time" stance recorded in Decision rule 1.
|
||||
|
||||
## Amendment note (2026-06-02): parse-time `ClickException` is JSON-wrapped under `--json`
|
||||
|
||||
The original Decision rule 1 stated that parse-time `ClickException` is
|
||||
*unchanged* and that "no JSON envelope is emitted in this case, because
|
||||
`handle_errors` has not yet been entered when the parser fires." That premise
|
||||
was correct about `handle_errors` — the per-command context manager genuinely
|
||||
never sees parse-time errors — but the conclusion no longer holds, because the
|
||||
CLI grew a **second, higher** boundary above `handle_errors`.
|
||||
|
||||
The root group (`SectionedGroup.main` in
|
||||
[`src/notebooklm/cli/grouped.py`](../../src/notebooklm/cli/grouped.py)) runs
|
||||
Click's superclass `main` in **non-standalone mode** specifically so it can
|
||||
catch the parse-time parser/callback `click.ClickException` that Click would
|
||||
otherwise render itself. Its docstring records the intent: catching the failure
|
||||
at "this root boundary [lets it] convert those failures once for every current
|
||||
and future subcommand option." When the raw argv contains `--json`
|
||||
(`_json_requested(args)`), the root group routes the exception through
|
||||
`_emit_json_click_error(exc)`, which calls `output_error(exc.format_message(),
|
||||
"VALIDATION_ERROR", json_output=True, exit_code=exc.exit_code)`. Concretely,
|
||||
under `--json`:
|
||||
|
||||
- The typed JSON envelope is emitted on stdout — `{ "error": true, "code":
|
||||
"VALIDATION_ERROR", "message": "<Click's formatted message>" }` — and nothing
|
||||
is written to stderr.
|
||||
- **Exit codes are preserved**, not normalised. The envelope passes
|
||||
`exit_code=exc.exit_code`, so a `UsageError` / `BadParameter` still exits `2`
|
||||
and a base `ClickException` still exits `1`. (This differs from the
|
||||
*post-parse* envelope path, which standardises on exit `1` per Decision rules
|
||||
2 and 4. The parse-time wrapper changes the *channel* — JSON instead of
|
||||
stderr — but not the exit code.)
|
||||
|
||||
Without `--json`, behavior is exactly as the original ADR described: the root
|
||||
group calls `exc.show()` (Click's native `Usage: ... / Error: ...` text on
|
||||
stderr) and exits with `exc.exit_code` (`2` for `UsageError` / `BadParameter`,
|
||||
`1` for the base `ClickException`). The `click.Abort` path is handled the same
|
||||
way at this boundary — `CANCELLED` envelope + exit `1` under `--json`, or
|
||||
`Aborted!` on stderr + exit `1` in text mode.
|
||||
|
||||
**Why the change.** The motivation is the same one that drove the original
|
||||
contract for post-parse errors: a JSON consumer that passes `--json` should
|
||||
never receive usage prose on stderr instead of a parseable envelope, even when
|
||||
the failure is an argv-level one (an invalid `--limit`, an unknown option, a
|
||||
missing required argument). Converting once at the root boundary gives every
|
||||
current and future subcommand a consistent envelope without per-command
|
||||
plumbing, and is strictly more useful to automation than the original
|
||||
"argv-level failures are structurally impossible to wrap" stance — which was
|
||||
true *inside Click's parser* but not at a boundary that runs the parser in
|
||||
non-standalone mode and catches what it raises.
|
||||
|
||||
**Source of truth.** This behavior is pinned by
|
||||
[`tests/unit/cli/test_json_validation_contract.py`](../../tests/unit/cli/test_json_validation_contract.py):
|
||||
`test_json_validation_errors_emit_json` asserts the `VALIDATION_ERROR` envelope
|
||||
on stdout (empty stderr, non-zero exit) for invalid limits/intervals/retries, a
|
||||
missing argument, an unknown option, and a root-callback validation failure;
|
||||
`test_text_validation_errors_keep_click_usage_output` asserts the unchanged
|
||||
text-mode path (exit `2`, `Usage:` on stderr, empty stdout); and
|
||||
`test_json_abort_emit_json` pins the `CANCELLED` envelope. The original
|
||||
Decision rule 1 below is retained for history; treat this note and the contract
|
||||
tests as authoritative wherever they diverge.
|
||||
|
||||
## Context
|
||||
|
||||
The CLI ships a stable error contract for automation: under `--json`, every
|
||||
fatal command path emits a *typed JSON error envelope*
|
||||
on stdout — a flat object of the shape
|
||||
|
||||
```json
|
||||
{ "error": true, "code": "<STABLE_CODE>", "message": "<human text>", ...extras }
|
||||
```
|
||||
|
||||
— and the process exits with the corresponding code from the table in
|
||||
[`docs/cli-exit-codes.md`](../cli-exit-codes.md). The canonical implementation
|
||||
is `_output_error(...)` in
|
||||
[`src/notebooklm/cli/error_handler.py`](../../src/notebooklm/cli/error_handler.py).
|
||||
|
||||
That contract was clear for library exceptions (`AuthError`, `RateLimitError`,
|
||||
`ValidationError`, ...). It was **not** clear for `click.ClickException` and
|
||||
its subclasses (`click.UsageError`, `click.BadParameter`, and bare
|
||||
`ClickException`), because `handle_errors(...)` deliberately re-raises them
|
||||
unmodified in its `except click.ClickException: raise` clause in
|
||||
[`src/notebooklm/cli/error_handler.py`](../../src/notebooklm/cli/error_handler.py).
|
||||
Click then renders its own `Usage: ... / Error: ...` prose to stderr and exits
|
||||
with its class-level `exit_code` (`2` for `UsageError`/`BadParameter`, `1` for
|
||||
the base `ClickException`).
|
||||
|
||||
There are two different phases in which `ClickException` subclasses are
|
||||
raised, and they behave identically at the Click level but mean very
|
||||
different things to a caller:
|
||||
|
||||
- **Parse-time** raises happen *before* the command body runs. Click's own
|
||||
parser raises `UsageError`/`BadParameter` when argv fails option/type
|
||||
validation (e.g. `--limit foo` where `--limit` is `IntRange`, an unknown
|
||||
flag, a missing required argument). The command function has not been
|
||||
invoked; `handle_errors(...)` is not yet on the stack; `--json` may have
|
||||
been *typed* on the command line but the parser never finished resolving
|
||||
it, so the caller's expectation of a JSON envelope is structurally
|
||||
impossible to honour from inside Click's parser.
|
||||
- **Post-parse** raises happen *inside* the command body or the service
|
||||
layer it calls, after argv parsing succeeded and after the `--json` flag's
|
||||
value is bound in `click.Context.params`. These are validation decisions
|
||||
the program itself made — flag-combination conflicts, computed
|
||||
preconditions, file-format checks — that happen to be expressed by
|
||||
raising a `ClickException` subclass instead of one of the library
|
||||
exception types. They reach the `except click.ClickException: raise`
|
||||
branch in `error_handler.py` and skip the envelope.
|
||||
|
||||
The CLI audit (the **P1#2
|
||||
"command-body `UsageError`/`BadParameter` bypass"** finding at lines 54-90)
|
||||
enumerated post-parse raise sites that ride this bypass:
|
||||
|
||||
- `src/notebooklm/cli/services/download.py:257` — `--force` / `--no-clobber`
|
||||
conflict
|
||||
- `src/notebooklm/cli/services/generate.py:394` — `--style custom` requires
|
||||
`--style-prompt`
|
||||
- `src/notebooklm/cli/research_cmd.py:158` — `--cited-only` requires
|
||||
`--import-all`
|
||||
- `src/notebooklm/cli/source_cmd.py:626` — `--cited-only` requires
|
||||
`--import-all`
|
||||
- `src/notebooklm/cli/chat_cmd.py:193` — `--new` mutually exclusive with
|
||||
`--conversation-id`
|
||||
- `src/notebooklm/cli/generate_cmd.py:57` — language-code validation
|
||||
|
||||
For each of these, `<cmd> ... --json` exits `2` with Click's usage text on
|
||||
stderr and **no JSON on stdout**. Automation branching on
|
||||
`json.loads(stdout)` (the published recipe in `docs/cli-exit-codes.md`)
|
||||
breaks. The meta-audit
|
||||
reframed the finding in three ways that matter here:
|
||||
|
||||
- **C1** declared the contract decision a stop-sentinel: no implementation
|
||||
PR should reshape these sites until the contract is decided and recorded,
|
||||
because two of the candidate fix shapes (route through the envelope
|
||||
vs. waive `ClickException` from `--json` entirely) drive contradictory
|
||||
patches.
|
||||
- **C3** widened the layer attribution: the same shape exists in service
|
||||
modules (`cli/services/generate.py:383, 394, 396, 398`,
|
||||
`cli/services/download.py:257, 259, 261`,
|
||||
`cli/services/source_mutations.py:18`), so any contract decision applies
|
||||
to both command and service code.
|
||||
- **I1** corrected the audit's claim that `docs/cli-exit-codes.md` was
|
||||
"internally ambiguous". The doc was **silent** on post-parse
|
||||
`UsageError`: line 52's table row describes Click re-raising its own
|
||||
exceptions, line 64's JSON section describes envelopes for library
|
||||
exceptions enumerated above, and the doc never crosses the two. This ADR
|
||||
fills that gap.
|
||||
|
||||
## Decision
|
||||
|
||||
Under `--json`, **every post-parse
|
||||
`click.ClickException`-subclass failure raised from a command body or from
|
||||
the service layer it calls emits the typed JSON error envelope** defined in
|
||||
[`docs/cli-exit-codes.md`](../cli-exit-codes.md), exits with the
|
||||
corresponding standard code, and writes no usage text to stderr.
|
||||
|
||||
Concretely:
|
||||
|
||||
1. **Parse-time `ClickException` is unchanged.** *(Superseded — see the
|
||||
"Amendment note (2026-06-02)" above. The root group now JSON-wraps
|
||||
parse-time `ClickException` under `--json`, preserving Click's exit code.
|
||||
The text below is retained for history and remains accurate only for the
|
||||
non-`--json` path.)* Click's parser keeps raising
|
||||
`UsageError` / `BadParameter` / `ClickException` for argv-level
|
||||
validation failures; Click keeps rendering its own
|
||||
`Usage: ... / Error: ...` text on stderr and exiting `2`
|
||||
(`UsageError` / `BadParameter`) or `1` (base `ClickException`). The
|
||||
re-raise in
|
||||
[`src/notebooklm/cli/error_handler.py`](../../src/notebooklm/cli/error_handler.py)
|
||||
stays. No JSON envelope is emitted *by `handle_errors`* in this case,
|
||||
because `handle_errors` has not yet been entered when the parser fires —
|
||||
but the root group (`SectionedGroup.main`) sits *above* `handle_errors`
|
||||
and does emit the envelope under `--json`; see the amendment note.
|
||||
2. **Post-parse `ClickException` flows through the typed envelope.**
|
||||
Command bodies and service-layer code that wish to signal a validation
|
||||
failure under the JSON contract MUST NOT raise
|
||||
`click.UsageError` / `click.BadParameter` / `click.ClickException`
|
||||
directly; they must instead call `output_error(...)` from
|
||||
`cli.error_handler` (or raise a library `ValidationError` /
|
||||
`ConfigurationError` that `handle_errors(...)` already maps onto the
|
||||
envelope). The natural code for a flag-combination or precondition
|
||||
conflict is `VALIDATION_ERROR` (exit `1`); the existing exit-code table
|
||||
in `docs/cli-exit-codes.md` lists the full mapping. No new error-code
|
||||
keys are introduced by this ADR.
|
||||
3. **The wire shape is unchanged.** Envelopes remain the flat object
|
||||
`{ "error": true, "code": "<CODE>", "message": "<text>", ...extras }`
|
||||
already produced by `_output_error`. Callers that already parse
|
||||
`json.loads(stdout)` and branch on `code` need no migration.
|
||||
4. **The text-mode contract is preserved.** When `--json` is *not* set,
|
||||
post-parse validation failures still surface a human-readable message
|
||||
on stderr and exit with the corresponding standard code. For sites
|
||||
refactored under rule 2, the message comes from `output_error(...)`
|
||||
rather than Click's `Usage: ... / Error: ...` formatter, so it omits
|
||||
the command-usage footer; this is the smallest user-visible delta
|
||||
consistent with rule 2. Marked sites (rule 5) keep Click's
|
||||
formatter unchanged.
|
||||
5. **Marked residual `ClickException` raises.** A small number of
|
||||
sites are correctly raising `ClickException` subclasses today and
|
||||
should keep doing so — they sit on input-validation boundaries before
|
||||
the command produces any output, and matching Click's own
|
||||
parser-style error rendering is the right UX (e.g. UTF-8 / file-read
|
||||
validation in `cli/input.py`, profile-name argument validation in
|
||||
`cli/profile_cmd.py`, entity-ID argument validation in `cli/resolve.py`,
|
||||
shared profile-name validation in `cli/services/login/profile_targets.py`).
|
||||
These are tracked exhaustively by inline marker comments:
|
||||
`# cli-input-validation: <reason>` for envelope-bypassing Click exceptions
|
||||
and `# cli-raw-exit: <reason>` for raw `SystemExit` sites outside
|
||||
`error_handler.py`. `tests/_guardrails/test_error_handler_allowlist.py`
|
||||
enforces that markers are present, non-empty, and not stale. New sites
|
||||
require a local marker with a justification; the default for any *other*
|
||||
post-parse validation failure is rule 2.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted:**
|
||||
|
||||
- `--json` is a reliable machine contract for every command-body failure.
|
||||
Automation branching on `json.loads(stdout)` and `code` works for
|
||||
validation conflicts the same way it already works for `RATE_LIMITED`
|
||||
and `AUTH_ERROR`.
|
||||
- `docs/cli-exit-codes.md` is no longer silent on post-parse `UsageError`,
|
||||
which closes meta-audit item I1 and removes a class of "but the docs
|
||||
said..." reports.
|
||||
- Command and service code converges on one error-emission path
|
||||
(`output_error(...)` / library exceptions through `handle_errors(...)`),
|
||||
which composes cleanly with the `cli/services/` extraction pattern in
|
||||
[ADR-0008](./0008-cli-services-extraction-pattern.md). Service modules no
|
||||
longer need to choose between raising Click exceptions (skips the
|
||||
envelope) and importing `output_error` (couples to the CLI layer); the
|
||||
ADR-0008 boundary becomes the right place to enforce typed-outcome
|
||||
returns. Service-layer convergence work referenced under meta-audit C3
|
||||
lands as separate per-site PRs that cite this ADR.
|
||||
- Tests can assert the envelope shape with the existing JSON-purity sweeps
|
||||
(`tests/unit/test_json_error_exit.py`, `tests/unit/test_json_stdout_purity.py`)
|
||||
by adding error-path argv cases for the previously-bypassed sites; no
|
||||
new test infrastructure is required.
|
||||
|
||||
**Unwanted:**
|
||||
|
||||
- Some post-parse failures previously rendered with Click's
|
||||
`Usage: ... / Error: ...` footer. Under rule 4, text-mode output for
|
||||
those sites loses the usage footer in exchange for a consistent stderr
|
||||
message. The audit considers this an acceptable trade because the
|
||||
usage-footer convention is for argv-shape errors, and post-parse
|
||||
failures are semantic-shape errors that already explain themselves in
|
||||
the message body.
|
||||
- Exit codes for post-parse failures shift from `2` (Click's
|
||||
`UsageError.exit_code`) to `1` (the standard `VALIDATION_ERROR` exit).
|
||||
This is intentional: `2` is reserved for system/unexpected errors and
|
||||
the parse-time path; a flag-combination failure is a user/application
|
||||
error and belongs at `1`. Scripts that branched on `?==2` after a
|
||||
post-parse `UsageError` would have been confusing "bad argv" with "bad
|
||||
combination" anyway; the new code is unambiguous.
|
||||
- The ADR creates a permanent split between parse-time and post-parse
|
||||
`ClickException` handling. Reviewers must apply rule 2 when adding new
|
||||
validation in command/service code, and reviewers must apply rule 1
|
||||
when changing Click parser configuration. The split is documented here
|
||||
so the next reviewer does not re-litigate it.
|
||||
- Pattern A and Pattern B service-layer sites enumerated by meta-audit C4
|
||||
(~10 modules) still need follow-up PRs to actually route through the
|
||||
envelope; this ADR records the contract but does not in itself move any
|
||||
code. Each follow-up PR cites this ADR and removes one site from the
|
||||
bypass.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Waive `ClickException` subclasses from `--json` entirely; document the
|
||||
exception.** Rejected. This freezes the audit-flagged behaviour as
|
||||
contractual: callers using `--json` for automation still cannot rely on
|
||||
JSON output for command-body validation failures, and the doc would
|
||||
have to enumerate which validations are "covered" and which are not.
|
||||
The matrix is unstable as new commands are added, and the audit's P1
|
||||
ranking specifically calls out automation breakage as the
|
||||
highest-impact user-facing issue (meta-audit C2).
|
||||
- **Convert every `ClickException` raise to a library `ValidationError`.**
|
||||
Rejected as a contract decision (but accepted as one valid *fix shape*
|
||||
per site). Some current `ClickException` sites are genuinely argv-level
|
||||
(the allowlisted ones in rule 5 of the Decision); forcing them through
|
||||
`ValidationError` would surface less-helpful messages to interactive
|
||||
users. The decision is "route through the envelope", not "stop using
|
||||
Click exceptions"; the allowlist exists precisely so each site can
|
||||
choose the right shape.
|
||||
- **Plumb `json_output` into every service helper and have each call
|
||||
`output_error(..., json_output=...)` directly from the helper.**
|
||||
Rejected as the *primary* contract shape (it would couple service
|
||||
modules to the CLI error layer, contradicting ADR-0008's boundary), but
|
||||
accepted as the smallest viable patch for any single site under
|
||||
remediation pressure. The preferred long-term shape is "service returns
|
||||
a typed outcome; command routes outcome through `output_error`"; the
|
||||
short-term patch is "service calls `output_error` directly". Both
|
||||
satisfy this ADR.
|
||||
- **Have `handle_errors` catch `ClickException` and convert to the
|
||||
envelope unconditionally.** Rejected *for `handle_errors`*. This would
|
||||
absorb parse-time `ClickException` too — but `handle_errors` is entered
|
||||
*inside* the command body, so the parse-time path never reaches it (Click
|
||||
renders parser errors via `BaseCommand.main` before invoking the command).
|
||||
Intercepting parse-time errors requires subclassing Click's command/group
|
||||
machinery, which was originally judged outside the scope of an
|
||||
error-handling contract change. The post-parse path is the only one
|
||||
`handle_errors` can address, and rule 2 already covers it via
|
||||
`output_error(...)`.
|
||||
|
||||
*(Update 2026-06-02: the "subclass Click's group machinery" approach was
|
||||
subsequently adopted — but at the root group, not in `handle_errors`.
|
||||
`SectionedGroup.main` runs Click's superclass in non-standalone mode and
|
||||
converts the parse-time `ClickException` to the JSON envelope under
|
||||
`--json`. This is a separate boundary from `handle_errors`, so the
|
||||
reasoning above — that `handle_errors` itself cannot see parse-time errors
|
||||
— still stands; see the amendment note.)*
|
||||
@@ -0,0 +1,70 @@
|
||||
# ADR-0016: Auth Identity and Core Logger Compatibility
|
||||
|
||||
## Status
|
||||
|
||||
Accepted.
|
||||
|
||||
## Context
|
||||
|
||||
The session-elimination work moved runtime ownership into
|
||||
`NotebookLMClient.__init__`, `ClientComposed`, and direct collaborators. Two
|
||||
historical invariants remain load-bearing after that move:
|
||||
|
||||
- `NotebookLMClient` and every auth-consuming collaborator must observe the
|
||||
same mutable `AuthTokens` instance. Auth refresh mutates token fields in
|
||||
place, so replacing the object for one consumer would strand the others on
|
||||
stale credentials.
|
||||
- The deleted `_core.py` module still has a logging compatibility footprint:
|
||||
downstream tests and user filters match logger name `"notebooklm._core"`.
|
||||
Logger names are strings, not imports, so deleting the module did not make
|
||||
the string safe to rename.
|
||||
|
||||
Both rules were previously explained partly in shipped source comments and
|
||||
partly in temporary implementation-plan notes. That made future maintenance
|
||||
fragile because the source had no stable rationale to cite.
|
||||
|
||||
## Decision
|
||||
|
||||
**Auth Instance Invariant.** `NotebookLMClient` keeps one authoritative
|
||||
`AuthTokens` object at
|
||||
`self._auth`, and all collaborators that need auth observe that object rather
|
||||
than a copied replacement. Refresh paths update the same object in place.
|
||||
|
||||
`CORE_LOGGER_NAME` remains the literal string `"notebooklm._core"` even though
|
||||
there is no active `notebooklm._core` compatibility module. Treat the string as
|
||||
a public-ish logging contract for filters, `caplog` assertions, and downstream
|
||||
diagnostics.
|
||||
|
||||
Source comments that need to explain either invariant should cite this ADR
|
||||
instead of temporary planning files.
|
||||
|
||||
## Consequences
|
||||
|
||||
Auth refresh remains identity-sensitive: construction may normalize
|
||||
`storage_path` by rebinding the incoming `AuthTokens` before collaborator
|
||||
construction, but once `self._auth` is set, runtime collaborators must alias
|
||||
that object. Tests that validate auth propagation should assert identity where
|
||||
identity is the contract, not only value equality.
|
||||
|
||||
Logging refactors must not rename the core logger casually. A future rename
|
||||
requires a compatibility plan that preserves or deliberately deprecates filters
|
||||
using `"notebooklm._core"`; ordinary module relocation is not enough rationale.
|
||||
|
||||
The ADR is intentionally narrow. It does not revive deleted compatibility
|
||||
modules, and it does not make `_core` an active import path.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Let each collaborator own an `AuthTokens` copy.** Rejected because auth
|
||||
refresh mutates credentials in place. Copies would make refresh visibility
|
||||
depend on which collaborator held which instance, creating stale-cookie and
|
||||
stale-CSRF bugs under concurrent use.
|
||||
|
||||
**Rename the logger to match the new module layout.** Rejected because the
|
||||
logger name has become an observable behavior independent of the deleted
|
||||
module. Renaming it would silently disable existing filters and test captures
|
||||
without improving runtime isolation.
|
||||
|
||||
**Leave the rationale only in source comments.** Rejected because the comments
|
||||
need a stable canonical home to cite. The invariant crosses multiple modules
|
||||
and should survive future source reshuffles.
|
||||
@@ -0,0 +1,116 @@
|
||||
# ADR-0017: Public-facade / private-implementation re-export convention
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (retroactive).
|
||||
|
||||
## Context
|
||||
|
||||
`notebooklm-py` exposes a public Python API that downstream callers import and
|
||||
depend on. Internally, the implementation is decomposed into many small,
|
||||
underscore-prefixed modules — the underscore-prefix privacy policy is recorded
|
||||
in [ADR-0012](0012-implementation-surface-convention.md), and the runtime
|
||||
decomposition into narrow collaborators in
|
||||
[ADR-0013](0013-composable-session-capabilities.md) /
|
||||
[ADR-0014](0014-feature-local-runtime-adapters.md). These goals pull against
|
||||
each other:
|
||||
|
||||
- Private modules are split, renamed, and merged frequently. The current
|
||||
module map in `docs/architecture.md` records moves such as the old
|
||||
`_session_contracts.py` home becoming `_runtime/contracts.py`.
|
||||
- Public import paths must stay stable: a renamed module that a caller imported
|
||||
from is a breaking change.
|
||||
|
||||
ADR-0012 establishes *that* a module is private (the leading underscore). It
|
||||
does not, on its own, say how a public symbol whose implementation lives in a
|
||||
private module is exposed. Without a paired convention, either the public
|
||||
surface ossifies the internal layout (blocking refactors) or a refactor
|
||||
silently breaks callers who imported from a private module.
|
||||
|
||||
The codebase already resolves this with a consistent pattern, but it is
|
||||
undocumented as a decision: short public modules that re-export from one or
|
||||
more underscore-prefixed implementation modules. `auth.py` is the most
|
||||
developed example — its body is almost pure re-exports forwarding to the
|
||||
`_auth/` package (see [ADR-0003](0003-auth-facade-write-through.md) and the
|
||||
`auth.py` row in `docs/architecture.md`).
|
||||
|
||||
## Decision
|
||||
|
||||
For each cohesive slice of the public API, a short, stable, **public** module
|
||||
re-exports the symbols whose **implementation** lives in one or more
|
||||
underscore-prefixed **private** modules. Callers import only from the public
|
||||
facade; the private modules are free to move.
|
||||
|
||||
The established facade ↔ implementation pairs (verified against the current
|
||||
source tree) are:
|
||||
|
||||
| Public facade | Private implementation |
|
||||
| ------------- | ---------------------- |
|
||||
| `types.py` | `_types/` package |
|
||||
| `config.py` | `_env.py` |
|
||||
| `urls.py` | `_url_utils.py` |
|
||||
| `io.py` | `_atomic_io.py` |
|
||||
| `log.py` | `_logging.py` |
|
||||
| `research.py` | `_research_task_parser.py` (plus public helpers) |
|
||||
| `auth.py` | `_auth/` package |
|
||||
|
||||
Single-purpose public helper modules (`artifacts.py`, `utils.py`,
|
||||
`migration.py`) follow the same shape: a stable public name backed by private
|
||||
collaborators.
|
||||
|
||||
The rules:
|
||||
|
||||
1. **Public modules are facades where practical.** A public module's body is
|
||||
usually re-exports (with at most a thin binding, e.g.
|
||||
`auth.enumerate_accounts`). Small public helper modules such as
|
||||
`artifacts.py`, `utils.py`, and `migration.py` may contain public helper
|
||||
logic directly when that is the stable API.
|
||||
2. **Callers import from the facade.** Importing from an underscore-prefixed
|
||||
module is unsupported and may break without notice.
|
||||
3. **Private modules may move freely.** Splitting, renaming, or merging a
|
||||
private module is *not* a breaking change as long as the facade's
|
||||
re-exports are preserved.
|
||||
4. **The facade is the compatibility contract.** Removing or renaming a symbol
|
||||
re-exported by a public facade *is* a breaking change; it is gated by the
|
||||
API-compat allowlist (`scripts/audit_public_api_compat.py` /
|
||||
`scripts/api-compat-allowlist.json`) and follows the deprecation strategy in
|
||||
[ADR-0018](0018-deprecation-strategy.md).
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted**
|
||||
|
||||
- **Refactor freedom**: internal decomposition (ADR-0012/013/014) proceeds
|
||||
without churning the public API.
|
||||
- **One obvious import path**: callers have a single stable location per
|
||||
concept, documented in `docs/python-api.md`.
|
||||
- **Mechanical compatibility checks**: because the public surface is a small,
|
||||
enumerable set of facades, `scripts/audit_public_api_compat.py` can diff it
|
||||
and fail the Code Quality gate on unapproved breaking changes.
|
||||
|
||||
**Unwanted**
|
||||
|
||||
- **Indirection**: reading a public module rarely shows the logic; you must
|
||||
follow the re-export into the private module.
|
||||
- **Drift risk**: a facade can fall out of sync with its implementation if a
|
||||
symbol is added privately but not re-exported. The public API compatibility
|
||||
audit (`scripts/audit_public_api_compat.py --check-stale`) and public-surface
|
||||
guardrails are the compatibility checks; `docs/architecture.md` is the
|
||||
human-readable module map.
|
||||
- **Two names for one thing**: every facade ↔ implementation pair is a small
|
||||
bookkeeping cost in the architecture/module map whenever a private module
|
||||
moves.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Expose the underscore modules directly and let callers import them.**
|
||||
Rejected. It couples every caller to the internal layout; the frequent
|
||||
private-module renames recorded in `docs/architecture.md` would each become a breaking
|
||||
change.
|
||||
- **One flat public module that contains the implementation.** Rejected. It
|
||||
forfeits the decomposition benefits of ADR-0013/014 (narrow, testable
|
||||
collaborators) and re-creates the god-module the runtime arc dismantled.
|
||||
- **No convention — decide ad hoc per symbol.** Rejected. Inconsistency is
|
||||
exactly what makes the public surface hard to audit mechanically; a single
|
||||
convention is what lets `audit_public_api_compat.py` reason about the whole
|
||||
surface.
|
||||
@@ -0,0 +1,99 @@
|
||||
# ADR-0018: Deprecation strategy (`_deprecation.py`)
|
||||
|
||||
> **Current state (2026-06-11).** The v0.8.0 runway described below completed:
|
||||
> `warn_get_returns_none`, `deprecated_kwarg`, `MappingCompatMixin`, and the
|
||||
> `NOTEBOOKLM_FUTURE_ERRORS` preview gate were removed when those breaking
|
||||
> changes became default behavior. `_deprecation.py` now exposes the generic
|
||||
> `warn_deprecated` helper plus the `NOTEBOOKLM_QUIET_DEPRECATIONS` suppression
|
||||
> gate. The only currently scheduled public deprecation is awaiting
|
||||
> `NotebookLMClient.from_storage(...)`; see `docs/deprecations.md`.
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (retroactive).
|
||||
|
||||
## Context
|
||||
|
||||
`notebooklm-py` evolves its public API: return shapes move from loose
|
||||
`dict[str, Any]` mappings to typed dataclasses (issue #1209), keyword
|
||||
arguments are renamed (e.g. `ResearchAPI.wait_for_completion`'s `interval` →
|
||||
`initial_interval`), and accessor semantics tighten (`get()` returning `None`
|
||||
on a miss will eventually raise a `*NotFoundError`, issue #1247). Each is a
|
||||
breaking change for some callers.
|
||||
|
||||
Shipping breaking changes with no runway strips downstream users of any chance
|
||||
to migrate; carrying every old behavior forever ossifies the API. The project
|
||||
needs a *single, consistent* way to (a) keep old behavior working for a
|
||||
deprecation window, (b) warn callers loudly enough to migrate, and (c) stay
|
||||
silent for tests and scripts that have already opted in. Without a single home
|
||||
for this, `warnings.warn(...)` calls scatter across feature modules with
|
||||
inconsistent messages and no shared suppression switch.
|
||||
|
||||
The mechanics already live in `src/notebooklm/_deprecation.py`; this ADR
|
||||
records the decision so the non-obvious versioning consequences are explicit.
|
||||
|
||||
## Decision
|
||||
|
||||
Centralize deprecation mechanics in `src/notebooklm/_deprecation.py`, gate
|
||||
**every** project `DeprecationWarning` behind the
|
||||
`NOTEBOOKLM_QUIET_DEPRECATIONS` environment variable, and name a concrete
|
||||
removal version whenever one is scheduled. The surviving reusable mechanism is
|
||||
`warn_deprecated(message, *, removal, stacklevel)`, used for one-off public
|
||||
deprecations such as awaiting `NotebookLMClient.from_storage(...)`.
|
||||
|
||||
Historical helpers from the v0.7 → v0.8 migration have been retired:
|
||||
`warn_get_returns_none`, `deprecated_kwarg`, and `MappingCompatMixin` no longer
|
||||
exist because `get()` now raises, `interval=` is gone, and typed returns are
|
||||
attribute-only dataclasses.
|
||||
|
||||
The rules:
|
||||
|
||||
1. **One module, one switch.** All deprecation warnings live in
|
||||
`_deprecation.py` and are silenced by `NOTEBOOKLM_QUIET_DEPRECATIONS`. No
|
||||
ad-hoc `warnings.warn(...)` scattered through feature modules.
|
||||
2. **Name the removal version when scheduled.** Every message states the version
|
||||
in which the old behavior is removed when one is known. Pass
|
||||
`removal=None` only for unscheduled deprecations or non-deprecation runtime
|
||||
warnings that deliberately do not route through `_deprecation.py`.
|
||||
3. **Warn at the boundary, not in the core.** Public methods warn; private
|
||||
helpers (e.g. `_get_or_none()`) already implement the future behavior
|
||||
without warning, so the eventual removal is a small, localized swap.
|
||||
4. **Retire runway helpers when the break ships.** Once a deprecation window
|
||||
closes, delete the compatibility helper instead of keeping dormant legacy
|
||||
behavior in `_deprecation.py`.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted**
|
||||
|
||||
- **Predictable runway**: callers get a named version and a consistent warning
|
||||
shape for every deprecation.
|
||||
- **Quiet for the already-migrated**: `NOTEBOOKLM_QUIET_DEPRECATIONS` lets
|
||||
tests and scripts opt out without monkeypatching `warnings`.
|
||||
- **Low-cost removal**: because the future behavior already lives in private
|
||||
helpers, dropping a deprecation is a small, localized edit.
|
||||
|
||||
**Unwanted**
|
||||
|
||||
- **Version coupling**: removal targets appear in messages and docs; bumping
|
||||
them requires a coordinated sweep.
|
||||
- **Two code paths during the window**: each active deprecation keeps both the
|
||||
old and new behavior alive until removal, adding temporary surface area.
|
||||
- **Discipline required**: the single-module rule only holds if contributors
|
||||
route new deprecations through `_deprecation.py` rather than inline warnings.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Clean breaks with no deprecation window.** Rejected for real public
|
||||
contracts: downstream callers need a runway, and the API-compat gate
|
||||
(ADR-0017 / `scripts/audit_public_api_compat.py`) exists precisely to stop
|
||||
unannounced breaks. (Clean breaks remain acceptable for non-contract
|
||||
internals and bug fixes — that is a separate policy, not a deprecation.)
|
||||
- **Per-feature `warnings.warn(...)` calls.** Rejected. It produces
|
||||
inconsistent messages, no shared removal-version vocabulary, and no single
|
||||
suppression switch — exactly the fragmentation this ADR prevents.
|
||||
- **Make `MappingCompatMixin` warn on every mapping access.** Rejected during
|
||||
the v0.7 runway. Warning on `get`/`in`/iteration would have flooded defensive
|
||||
callers who were already forward-compatible; warning only on `__getitem__`
|
||||
targeted the access pattern that actually broke when the dict return became a
|
||||
dataclass. The mixin has since been removed.
|
||||
@@ -0,0 +1,226 @@
|
||||
# ADR-0019: Error-and-return contract for the public API
|
||||
|
||||
## Status
|
||||
|
||||
Accepted and implemented in v0.8.0. The additive enforcement floor landed in
|
||||
v0.7.0, and the breaking flips this ADR queued shipped in v0.8.0: namespace
|
||||
`get()` methods raise their `*NotFoundError`, `get_or_none()` is the sanctioned
|
||||
`None`-on-miss lookup, dict-subscript compatibility was removed, deprecated
|
||||
keyword aliases were removed, and synchronous kickoff refusals now raise.
|
||||
|
||||
## Context
|
||||
|
||||
The public API has accreted incompatible conventions for the *same* outcome.
|
||||
Two grounded surveys found a single concept — "not found" — encoded **eight**
|
||||
ways (raise `*NotFoundError`; `None`+warn; silent `None`; empty sentinel object;
|
||||
`""`; a `"not_found"` status string; `ValueError`; silent no-op `None`),
|
||||
synchronous server refusal **three** ways, and a null/shape-drift result
|
||||
**five** ways. The divergence is *growing*: `mind_maps` (newest namespace,
|
||||
#1256) adopted the about-to-be-deprecated None-on-miss convention — without even
|
||||
the deprecation warning — and reached for `ValueError`.
|
||||
|
||||
Related decisions: [ADR-0005](0005-idempotency-taxonomy.md) (mutating-RPC retry
|
||||
policy), [ADR-0011](0011-schema-validation-policy.md) (strict-decode default;
|
||||
shape-drift raises `UnknownRPCMethodError`/`DecodingError` — this ADR extends
|
||||
that boundary to the hand-rolled list helpers ADR-0011 left for follow-up),
|
||||
[ADR-0012](0012-implementation-surface-convention.md) (private surface),
|
||||
[ADR-0017](0017-public-facade-private-implementation.md) (the public facade
|
||||
surface owns the compatibility contract; logic stays in private modules),
|
||||
[ADR-0018](0018-deprecation-strategy.md) (how breaking changes ship). None of
|
||||
them says *what a method returns versus raises for each failure mode* — that gap
|
||||
is what this ADR closes, making the committed Class-1 work (#1247 flip `get()`
|
||||
to raise; #1254 remove `interval`; #1251 drop `MappingCompat`) instances of one
|
||||
contract and converging the rest in the same release.
|
||||
|
||||
## Decision
|
||||
|
||||
> **A return value encodes only success and genuine asynchronous-lifecycle
|
||||
> state. *Resource* absence is an exception (`*NotFoundError`); the *poll-observed*
|
||||
> absence of an in-flight task is a typed lifecycle status, not an error. Server
|
||||
> refusals, shape-drift, and transport faults are always exceptions. `None` is
|
||||
> reserved for the cases enumerated below — it, `""`, `ValueError`, and an
|
||||
> untyped `"not_found"` string are never used to signal that an error happened.**
|
||||
|
||||
### Return-value vocabulary (allowed meanings)
|
||||
|
||||
| Shape | May mean |
|
||||
| ----- | -------- |
|
||||
| object / dataclass | success |
|
||||
| collection | zero-or-more (empty → `[]`) |
|
||||
| status handle (`GenerationStatus`/`ResearchTask`/`ResearchStart`) | async lifecycle only; terminal `failed`/`removed` and the *poll-observed* `not_found` are typed states. `status="failed"` ⇒ *started-then-failed*, never *couldn't-start* |
|
||||
| `None` | (a) idempotent `delete`; (b) explicit `get_or_none()`; (c) no-payload **command** success (`update`, `configure`, `remove_from_recent`, `rename(return_object=False)`); (d) a transient *not-ready* read (`get_tree` of an existing-but-unpopulated map); (e) a domain-optional field |
|
||||
|
||||
Anything else that today carries an error meaning is banned.
|
||||
|
||||
### Contract by operation class
|
||||
|
||||
| Class | Methods | Contract |
|
||||
| ----- | ------- | -------- |
|
||||
| Lookup one | `get` | found → object; missing → **raise `*NotFoundError`**. Public `get_or_none()` is the sole sanctioned `None`-on-miss path. |
|
||||
| List many | `list`, `list_*` | always a collection; empty → `[]`. |
|
||||
| Derived read | `get_summary`, `get_description`, `get_guide`, `get_tree`, `check_freshness` | **do not police parent existence** — missing parent → empty / not-ready value (`""`, empty dataclass, `None` tree); shape-drift → **raise** (`DecodingError`/`UnknownRPCMethodError`). Resource existence is `get()`'s job, not a derived read's. |
|
||||
| Idempotent mutation | `delete` | success *or* already-absent → `None`; raise only on real failure. |
|
||||
| Mutate existing | `rename`, `update`, `configure` | target missing → **raise `*NotFoundError`**; no-payload success → `None`. |
|
||||
| Async kickoff | `generate_*`, `create`, `revise_slide`, `retry_failed`, `research.start`, `mind_maps.generate` | accepted → return status handle; **synchronous refusal → raise**; null/missing-id/shape-drift → raise. |
|
||||
| Lifecycle status / await | `poll_status`, `research.poll`, `wait_for_completion` | reflect lifecycle; terminal `failed`/`removed` stay returned status; *poll-observed* `not_found` is a typed sentinel (not a raise); does **not** raise for a terminal `failed`, but **does** raise on timeout and on cross-cutting faults. |
|
||||
| Readiness wait | `wait_until_ready`, `wait_for_sources` | return the ready **resource**; raise `*TimeoutError` on timeout and the domain error on terminal processing failure. (*Distinct from the lifecycle-status handles above.*) |
|
||||
| Cross-cutting | any | transport→`NetworkError`/`RPCTimeoutError`; auth→`AuthError`; rate-limit→`RateLimitError`; oversize→`RPCResponseTooLargeError`; decode→`DecodingError`. Always raise. |
|
||||
|
||||
The load-bearing line: **resource-absent / couldn't-start / timed-out → raise; started-then-reached-a-terminal-state, or a transient poll observation → data.** A synchronous refusal is *couldn't start* (raise); a polled `failed` is *started then failed* (data); a poll that doesn't yet see an accepted task is a transient *not_found* (typed status), categorically different from looking up a resource that does not exist (raise).
|
||||
|
||||
Absence detection is single-sourced where shared (e.g. `_detect_kind` for mind maps): the detector raises `*NotFoundError`, and each operation class *interprets* that one signal — a derived read swallows it to empty/`None`, a mutate-existing re-raises, an idempotent `delete` swallows it to `None`. One detector, three contracts, no per-method re-deciding.
|
||||
|
||||
### Exception taxonomy
|
||||
|
||||
Ratify the existing tree (`NotebookLMError` root; multi-base
|
||||
`*NotFoundError(NotFoundError, RPCError, <Domain>Error)`; the `RPCError`
|
||||
transport subtree; `WaitTimeoutError(…, TimeoutError)`). `NoteError` /
|
||||
`NoteNotFoundError` and `MindMapError` / `MindMapNotFoundError` have landed,
|
||||
mirroring `SourceNotFoundError`. `ArtifactTimeoutError` now inherits
|
||||
umbrella-first from `WaitTimeoutError` before `ArtifactError`. No new "refusal"
|
||||
exception — refusal reuses the existing `RateLimitError`/`RPCError`.
|
||||
|
||||
### Rules
|
||||
|
||||
1. **Resource absence raises.** `get()` raises `*NotFoundError`; `get_or_none()`
|
||||
is the only sanctioned `None`-on-miss path and must re-raise anything that is
|
||||
not a genuine miss. (Poll-observed task absence is *not* resource absence —
|
||||
see Rule 4.)
|
||||
2. **Refusal raises.** A synchronous `USER_DISPLAYABLE_ERROR` propagates as the
|
||||
`RateLimitError`/`RPCError` the transport layer raises. The old kickoff
|
||||
behavior that swallowed refusal into `GenerationStatus(status="failed")` or
|
||||
synthesized `failed` for a missing artifact id was removed in v0.8.0;
|
||||
kickoff refusal now raises, and missing/degenerate ids raise
|
||||
`DecodingError`/`ArtifactFeatureUnavailableError`.
|
||||
3. **Drift raises.** A malformed/unparseable RPC payload raises
|
||||
`DecodingError`/`UnknownRPCMethodError` ([ADR-0011](0011-schema-validation-policy.md));
|
||||
it is not collapsed to `None`/`""`/`[]`/a sentinel. v0.8.0 tightens the
|
||||
**positional shape-drift** collapse in the hand-rolled list helpers
|
||||
(`_note_service.py:135`, `_artifact/listing.py:113`). The composite-lister
|
||||
`except RPCError`/`HTTPError` that returns *partial* studio artifacts when the
|
||||
mind-map sub-fetch is down (`_artifact/listing.py:126-138`) is a **deliberate
|
||||
partial-availability** behavior, **not** drift-collapse — it is out of scope
|
||||
for Rule 3 and decided separately (see Scope).
|
||||
4. **Lifecycle is data.** Async status handles carry `failed`/`not_found`/
|
||||
`removed` as typed states; `wait_for_completion` returns a terminal `failed`
|
||||
and raises only on timeout or a cross-cutting fault. The poll-observed
|
||||
`not_found` (artifact not yet listed, or research task absent) is a typed
|
||||
sentinel — `GenerationStatus.is_not_found`, and a **new** `ResearchStatus.NOT_FOUND`
|
||||
member (distinct from the existing `NO_RESEARCH` "nothing in flight"). The
|
||||
*termination* guarantee for a task that never appears lives in
|
||||
`wait_for_completion`, not `poll_status`: a sustained run of `not_found`
|
||||
(`max_not_found`/`min_not_found_window`) escalates to a terminal `removed`
|
||||
status (`_artifact/polling.py:366-384`). `poll_status` is a stateless
|
||||
primitive where `not_found` is inherently *lag-or-bogus* ambiguous by design;
|
||||
callers needing a terminal answer use `wait_for_completion`.
|
||||
5. **The facade owns the contract.** Per [ADR-0017](0017-public-facade-private-implementation.md)
|
||||
the public facade *surface* owns the compatibility contract (logic stays
|
||||
private); breaks ship via [ADR-0018](0018-deprecation-strategy.md). The
|
||||
#1247/#1254/#1251 breaks had a v0.7.0 deprecation runway, while the
|
||||
refusal/`ValueError`/`update` changes were deliberate clean breaks in the
|
||||
already-breaking v0.8.0 and were allowlisted
|
||||
(`scripts/api-compat-allowlist.json`). Idempotency is unchanged
|
||||
([ADR-0005](0005-idempotency-taxonomy.md): kickoffs stay non-blind-replayable).
|
||||
|
||||
`ValueError` remains valid for **input validation**; it is banned only for
|
||||
resource absence and server failure.
|
||||
|
||||
**Retry guidance.** Because `*NotFoundError` multi-inherits `RPCError` (see the
|
||||
exception taxonomy), transport-retry code must catch the *narrow* transport
|
||||
exceptions — `NetworkError`/`RPCTimeoutError`/`RateLimitError` — and never the
|
||||
broad `RPCError`, so a retry loop never silently swallows a `*NotFoundError`.
|
||||
|
||||
## Scope
|
||||
|
||||
*(This `Scope` section and the `Enforcement` section below intentionally extend the standard six-section ADR template; both carry convergence-specific load — see ADR review thread.)*
|
||||
|
||||
In scope: the operation classes above across `notebooks`, `sources`,
|
||||
`artifacts`, `chat`, `research`, `notes`, `mind_maps`, `sharing`, `settings`.
|
||||
Explicitly **deferred / follow-existing-contract** (not changed in this ADR;
|
||||
tracked separately): bulk/derived helpers that today swallow drift to empty
|
||||
data (`notebooks.get_metadata`/`get_source_ids`/`get_raw`, the research-task
|
||||
parser fallbacks); `share`; export/download paths; and the chat surface. The
|
||||
composite-lister partial-availability policy (Rule 3) is decided in its own PR.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted**
|
||||
|
||||
- One predictable rule per operation class; new features stop re-deciding.
|
||||
- Type-narrowing works: `get()` returns a non-optional object; callers branch on
|
||||
exceptions, not on `None`/`""`/sentinel ambiguity.
|
||||
- A refusal can no longer masquerade as a started-then-failed task.
|
||||
- Mechanically auditable alongside [ADR-0017](0017-public-facade-private-implementation.md).
|
||||
|
||||
**Unwanted**
|
||||
|
||||
- A large v0.8.0: not-found + refusal + null + reads + mutations land together.
|
||||
Mitigated by per-wave verification and the `api-compat`/golden-fixture gates.
|
||||
- A small extra cost only on `rename`'s default path (it re-fetches to return the
|
||||
renamed object; `return_object=False` is the existing opt-out that avoids it).
|
||||
Derived reads add **no** existence RPC — they return empty on a missing parent.
|
||||
- Callers relying on `GenerationStatus(status="failed")` for rate-limit handling
|
||||
must catch `RateLimitError`; the public `with_rate_limit_retry` helper is
|
||||
rewritten accordingly.
|
||||
- The convention must be *enforced*, not just documented, or it re-accretes (it
|
||||
already did, in `mind_maps`). Enforcement is in scope for 0.8.0 — see below.
|
||||
|
||||
## Enforcement (in scope for 0.8.0)
|
||||
|
||||
The 8-way divergence happened because consistency was enforced only by review,
|
||||
not by types — `mind_maps` re-diverged the moment it was added. A documented
|
||||
contract that nothing checks will re-accrete, so 0.8.0 lands a **tiered
|
||||
enforcement floor**:
|
||||
|
||||
- **Tier 1 — conformance test (mandatory).** A parametrised
|
||||
`test_public_api_contract.py` asserts the contract as a **static shape** check
|
||||
over the *whole* public surface via its **own** `inspect.signature().return_annotation`
|
||||
walk over all namespaces (incl. `mind_maps`, which the `audit_public_api_compat.py`
|
||||
collector under-covers; not its comparator, which ignores return types): every
|
||||
namespace `get(...)` has a non-Optional return **and** a paired
|
||||
`get_or_none(...)` returning Optional;
|
||||
`delete(...) -> None`; no public lookup is annotated `X | None`. Deferred
|
||||
*behaviours* (see Scope) are carried in an explicit, reason-tagged **exemption
|
||||
allowlist** (same idiom as `api-compat-allowlist.json`) so every gap is visible
|
||||
and shrinking, never silent. The divergence that occurred
|
||||
(`mind_maps.get() -> MindMap | None`) is a signature smell this catches with no
|
||||
backend.
|
||||
- **Tier 2 — single-sourced lookup logic (mandatory, structure-first).** A shared
|
||||
`unwrap_or_raise(obj, exc)` helper backs each namespace's own **fully-typed**
|
||||
`get`/`get_or_none` (`get()` = `unwrap_or_raise(await self._fetch_one(...),
|
||||
<Resource>NotFoundError(...))`). This is **PR #1**: the lookup *logic* is
|
||||
single-sourced while signatures stay per-class. A generic `ResourceAPI[T]`
|
||||
*base* was considered and **rejected** (momus 2/3) — a `*ids` base erases
|
||||
public-signature typing (the namespaces differ in arity) and `delete` is
|
||||
irreducibly per-namespace (`mind_maps.delete(..., kind=...)` is non-idempotent
|
||||
+ kind-dispatched), so `delete` stays per-namespace.
|
||||
- **Tier 3 — sealed async result types (resolved #1345: rejected).** Replacing
|
||||
`GenerationStatus` with a sealed/discriminated result was evaluated and
|
||||
**rejected**. The load-bearing overload it targeted — a synchronous
|
||||
*couldn't-start* masquerading as `status="failed"` — was removed by Tier 1
|
||||
(#1342 makes refusals raise), so a returned `failed` now means only
|
||||
*started-then-failed*. The residual `not_found`/`removed`/rate-limit juggling
|
||||
is poll-loop interpretation and adapter projection. The non-breaking follow-up
|
||||
did land as `GenerationState(str, Enum)` for `GenerationStatus.status`,
|
||||
mirroring `ResearchStatus`. If sealed types are ever revisited, introduce them
|
||||
via parallel `poll_result()`/`wait_result()` APIs rather than breaking the
|
||||
existing ones in place.
|
||||
|
||||
Tier 1 + Tier 2 are required for 0.8.0; together they make this contract
|
||||
type/CI-enforced rather than review-enforced.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Keep `status="failed"` for synchronous refusal** (local consistency).
|
||||
Rejected — deepens the soft-fail-as-data pattern #1247 is leaving; makes
|
||||
`retry_failed` the outlier.
|
||||
- **Make `wait_for_completion` raise on terminal failure**, for symmetry with
|
||||
timeout. Rejected — a terminal `failed` is real async data; only
|
||||
*couldn't-start*, *timeout*, and cross-cutting faults are exceptional.
|
||||
- **Raise on a poll-observed unknown task.** Rejected — a poll loop cannot treat
|
||||
replication-lag as exceptional; a typed sentinel is the right shape.
|
||||
- **Force `get_summary`/`get_description` to raise on a missing parent.**
|
||||
Rejected — no parent-existence signal; would mean an extra RPC per call for a
|
||||
rare error; empty stays a legitimate domain value there.
|
||||
- **Split the convergence across 0.8.0 + 0.9.0.** Considered for safety;
|
||||
maintainer chose all-in to fix the divergence once.
|
||||
@@ -0,0 +1,56 @@
|
||||
# ADR-0020: Sealed async result types for artifact generation
|
||||
|
||||
## Status
|
||||
|
||||
Proposed. **Baseline: `main` at #1447 (`f0d2d1be`, v0.8.0)** — `GenerationStatus.status` is already typed `GenerationState(str, Enum)`, with the private `_status_from_code` code→state normaliser and a *test-enforced* poll/wait partition (`tests/unit/test_generation_state.py::test_poll_status_never_returns_removed`). This ADR is the "own ADR" that [ADR-0019](0019-error-and-return-contract.md) Tier 3 — amended by #1446 to "sealed union **rejected for 0.8.0**; if ever revisited, via parallel APIs" — pointed to. It records the sealed design and recommends **continued deferral**. (Code references below use stable symbol/qualified-path names rather than line numbers, which drift.)
|
||||
|
||||
## Context
|
||||
|
||||
`GenerationStatus` (`_types/artifacts.py`) is one non-frozen `@dataclass` (`task_id, status: GenerationState, url: str|None, error: str|None, error_code: str|None, metadata`) returned by **~13 methods** in two roles:
|
||||
|
||||
- **Snapshot** (`generate_*` ×10, `revise_slide`, `retry_failed`, `poll_status`): point-in-time — `pending|in_progress|completed|failed|not_found|unknown`. Kickoff methods take no `wait` flag, so their return is role-stable.
|
||||
- **Terminal** (`wait_for_completion`): `completed|failed|removed`. Timeout *raises* `ArtifactTimeoutError`.
|
||||
|
||||
The partition (`not_found` poll-only, `removed` wait-only) holds by **producer convention** and is **test-enforced** (`test_poll_status_never_returns_removed`), but not **type-enforced**. Residual problems after #1447: (1) illegal field combinations are still representable (all of `url/error/error_code` optional); (2) the partition isn't in the type; (3) `GenerationStatus.is_rate_limited` derives from `error_code == "USER_DISPLAYABLE_ERROR"` **or** a fragile error-message substring fallback. #1342 already removed the load-bearing overload (couldn't-start raises), so this is type hygiene, not a correctness gap.
|
||||
|
||||
## Decision
|
||||
|
||||
### A. Design of record (the shape if/when built)
|
||||
|
||||
**A1 — Two role types.** `PollResult` = `Pending|InProgress|Completed|Failed|NotFound|Unknown` (snapshot methods); `WaitOutcome` = `Completed|Failed|Removed` (the waiter). Encodes the partition in the type.
|
||||
|
||||
**A2 — Separate `@dataclass(frozen=True)` variants, NOT subtypes of `GenerationStatus`.** Rationale: a *frozen* variant cannot subclass the *non-frozen* `GenerationStatus`, and a mutable subtype cannot make state immutable; field-ordering (required-after-defaulted) is sidesteppable with `kw_only=True` but immutability is not. **Caveat that shrinks the prize:** the headline benefit — required per-variant fields — is *largely unachievable against today's data*: `Completed.url` is legitimately `None` for non-media artifacts (`ArtifactRow.is_media_ready` returns `True` for non-media types regardless of URL) and for kickoff-completed (`_parse_generation_result` in `_artifacts.py`); `Failed.error` is sometimes `None` (`poll_status` construction in `_artifact/polling.py`); `Removed` carries `error` but no `error_code` (the wait loop's removed-construction in `_artifact/polling.py`). So `Completed.url`/`Failed.error` stay `str | None` (or `Completed` splits into media-completed `url: str` vs document-completed `url: str|None`). The variants thus buy *role separation + exhaustive `match` + structured failure*, **not** illegal-states-unrepresentable. **Before committing to A1–A6, re-evaluate the subtype-refinement alternative below: given this finding, it likely delivers more value per cost.**
|
||||
|
||||
**A3 — Timeout stays an exception** (no `TimedOut` variant). Consistent with ADR-0019.
|
||||
|
||||
**A4 — `failure_reason: FailureReason` (`RATE_LIMIT | OTHER | UNKNOWN`) on `Failed`/`Removed`.** Derivation must be pinned: `error_code == "USER_DISPLAYABLE_ERROR"` (or the current message heuristic, *retained inside the classifier* because `error_code` is often absent) → `RATE_LIMIT`; a present-but-non-rate-limit failure → `OTHER`; an unclassifiable failure (no `error_code`, no message match) → `UNKNOWN`. A4 does **not** get to drop the substring matching for free — it relocates it into a single classifier. `error`/`error_code` remain optional on the variants.
|
||||
|
||||
**A5 — Compatibility is duck-typed/source-level only.** A shared `Protocol`/mixin exposing `.status` + `.is_*` lets *predicate* consumers (the `hasattr(status, "is_complete")` duck-type in `cli/services/artifact_generation.py`; `_artifact/polling.py`'s `is_complete or is_failed`) and `match`/`.status` callers migrate incrementally. It does **NOT** preserve nominal checks: the `isinstance(result, GenerationStatus)` gates in `cli/services/artifact_generation.py`, the direct JSON field-mirroring in `cli/artifact_cmd.py`, `wait_for_completion`'s `on_status_change` callbacks, and `ArtifactTimeoutError.status_transitions` (`exceptions.py`) all reference the concrete type and are **explicit migration work**. The frozen variants also **drop the str-Enum's raw-string-construction tolerance** (`GenerationStatus(status="completed")` is valid today; a frozen union is not) — an intended behavior change to call out.
|
||||
|
||||
**A6 — Additive-first migration, with an honest gap.**
|
||||
- **Phase 1 (additive):** add `poll_result()` / `wait_result()` returning the variants, implemented via a `GenerationStatus → variant` adapter that **`match`es on the constructed `.status`** (the `GenerationState` member). `_status_from_code` is *not* the seam — it is the upstream int→state normaliser and never emits `NOT_FOUND`/`REMOVED` (those are constructed directly in `_artifact/polling.py`). The ~13 flat methods keep returning `GenerationStatus`.
|
||||
- **Gap (no cheap additive pair for kickoff):** the 12 snapshot *kickoff* methods (`generate_*`, `revise_slide`, `retry_failed`) cannot each get a `*_result()` twin without 12 new methods. Options: (a) a public `GenerationStatus.as_poll_result()` converter callers opt into; (b) accept that kickoff return-flipping is a **Phase-3 breaking change only**. The ADR picks (a) as the cheaper bridge.
|
||||
- **Phase 2 (runway):** deprecate the flat-returning methods (ADR-0018 runway, real contract); migrate the CLI/services/docs and the **105 `GenerationStatus(...)` construction sites across 15 test files**.
|
||||
- **Phase 3 (one breaking flip, at a major):** remove/re-annotate the flat path + flip kickoff returns. Re-annotating in place is `changed-return`-flagged by `scripts/audit_public_api_compat.py`, so it only happens here.
|
||||
|
||||
### B. Timing decision — defer (strengthened)
|
||||
|
||||
**Adopt A1–A6 as design of record, but do NOT build it now.** Both the original rationale and the review findings point the same way, more strongly than v1 claimed: the load-bearing overload is already gone (#1342); the str-Enum + test-enforced partition captured the realizable cheap value; and **the headline prize (required per-variant fields) is mostly unachievable against real data (A2)** — so the breaking full-split now buys only role-separation + exhaustive `match` + structured `failure_reason` over the *non-breaking* subtype refinement. That margin does not justify a multi-phase program across 13 methods, the CLI, callbacks/exceptions, and 105 test sites right after v0.8.0. **If a trigger fires, start by re-evaluating the subtype refinement (Alternatives) — A1–A6 is the spec only if that lighter path is rejected.**
|
||||
|
||||
**Revisit triggers (build when any holds):** (a) a concrete recurring bug from the flat shape; (b) a planned major / version-runway window already open; (c) a feature needing per-variant fields; (d) a research-side convergence — `ResearchStatus` is already a `str, Enum` with `NOT_FOUND`, so a *shared* sealed-result pattern could be built once and amortised across both lifecycles (the project values building a pattern with its gate once over per-namespace re-deciding).
|
||||
|
||||
## Scope
|
||||
|
||||
In: the type shape, role split, timeout/`failure_reason`/compat decisions, migration sequence. Out: the exception model, the `GenerationState` enum (#1447, done), `Source`/`Research` status types, and — under (B) — implementation. (Like ADR-0019, this ADR carries an explicit `Scope` section.)
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Deferred (recommended):** zero new code; the design is recorded so it is neither re-litigated nor accidentally owed; supersedes ADR-0019's "Tier 3 deferred." The #1447 groundwork keeps a future Phase 1 cheap.
|
||||
- **If built:** exhaustive `match`, role-distinct poll/wait types, `failure_reason` consolidated into one classifier. **Not** delivered: illegal-states-unrepresentable (fields stay optional, A2). Costs: a dual surface during the runway; migration of the CLI/services/callbacks/exceptions/105 test sites; lost raw-string construction; one breaking flip at a major.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Status quo — typed-flat `GenerationStatus` (recommended resting state).** Accepted as the deferral baseline.
|
||||
- **Subtype refinement (`Completed(GenerationStatus)` …).** Non-breaking, covers all 13 methods (annotations stay `-> GenerationStatus`), gives `isinstance`/`match` + variant methods. Given A2 shows required fields are unachievable anyway, **this is the stronger candidate *if* the trigger ever fires** — it delivers the realizable benefits (role discrimination, `match`) without the runway or the breaking flip. Its only loss vs separate unions (true immutability + required fields) is the part that doesn't hold against the data. The Decision (A1–A6) specs the full split for completeness, but implementation should re-confirm this path is not the better choice first.
|
||||
- **Re-annotate methods in place.** Rejected — `changed-return` break, no runway.
|
||||
- **`TimedOut`/`RateLimited` as variants.** Rejected — timeout is exceptional (A3), rate-limit is failure-detail (A4).
|
||||
@@ -0,0 +1,43 @@
|
||||
# ADR-0021: Transport-neutral application layer (`_app/`)
|
||||
|
||||
## Status
|
||||
|
||||
Accepted — realized on the `refactor/cli-business-logic` prototype branch and adopted on main. Supersedes, on the question of *placement*, the earlier "wire transport-neutral utilities **in place**" proposal: that proposal's in-place framing is withdrawn in favour of relocation into a dedicated package.
|
||||
|
||||
## Context
|
||||
|
||||
The CLI layer (`cli/`) carried business logic — id validation/resolution, plan-building, status projection, retry/wait orchestration, junk-source detection, content selection, `auth check`/`doctor` diagnostics — entangled with Click/Rich presentation. ADR-0008 (CLI-services extraction) moved much of it into `cli/services/`, but those services kept **signature-level** Click coupling (`json_output` flags, `Console`/`raw_args`/`emit_status` seams). Additional front-ends — the FastMCP server and the FastAPI REST server — therefore needed the logic without importing Click.
|
||||
|
||||
Two independent oracle reviews (Claude + Codex) converged on relocating the neutral core out of `cli/` rather than wiring utilities in place; momus (Claude + Codex) cleared the resulting plan after a rev pass. This ADR records the decision.
|
||||
|
||||
## Decision
|
||||
|
||||
Relocate transport-neutral **business logic** into `src/notebooklm/_app/` (underscore-private per ADR-0012). CLI / MCP / REST are thin sibling **adapters** over it.
|
||||
|
||||
```
|
||||
client.* (public domain API) → _app/ (neutral) → cli/ (Click) + mcp/ (FastMCP) + server/ (FastAPI)
|
||||
```
|
||||
|
||||
- **Contract.** Per verb: typed frozen-dataclass `Request`/`Plan`/`Result`; a pure `build_<verb>_plan(req) -> Plan` (raises the public `notebooklm.exceptions` hierarchy on bad input); `async execute_<verb>(plan, client, *, progress: ProgressSink | None) -> Result`. The adapter parses its own inputs into the request, calls the neutral core, and renders the typed result into its own envelope vocabulary — the CLI builds the byte-stable `--json` envelope per ADR-0015.
|
||||
- **Boundary.** `_app/` imports no transport framework (`click`, `rich`, `fastmcp`, `fastapi`, `starlette`, `uvicorn`), no adapter sibling (`notebooklm.cli`, `notebooklm.server`), no RPC runtime sibling (`notebooklm.rpc`), and no private `notebooklm._*` runtime sibling outside `_app` itself (enforced by `tests/_guardrails/test_app_boundary.py`). Rich/Click/FastMCP/FastAPI-coupled collaborators (consoles, prompters, spinners, importers, resolvers, request/response objects) are **injected** as callables/Protocols.
|
||||
- **Errors.** `_app` raises the public `notebooklm.exceptions` hierarchy for library/domain failures; local deterministic adapter-input exceptions are either public-base subclasses or caught and translated before crossing the adapter boundary. Classification is centralized in `_app.errors.classify(exc) -> ClassifiedError` — the single neutral source of the failure **category** decision (per ADR-0019). Each adapter keeps its **own** code vocabulary and projects the category onto it: the CLI `error_handler` onto string codes + exit codes, the MCP server onto its manifest-pinned codes, and the REST server onto HTTP status + `{"error": {"category": "...", "message": "..."}}`. Consistency gates fail CI if adapter projections drift from `classify`. No adapter envelope-building lives in `_app` (no `.payload` / `to_envelope`); genuinely transport-neutral shaping shared by adapters is hoisted into `_app.serialize` (`to_jsonable`, `source_summary`).
|
||||
- **Patch-seam discipline.** Command modules are **not** moved; anything a test stubs is read at call time or injected — never closed over at import (the trap documented at `tests/unit/cli/conftest.py`). *Amended #1481 (2026-06-08):* the per-command `patch("...<x>_cmd.NotebookLMClient")` client seams are **retired**. Command bodies resolve the client via `cli.auth_runtime.resolve_client_factory(ctx)`, injected through Click's `ctx.obj["client_factory"]`; tests substitute a fake with `runner.invoke(..., obj=inject_client(mock_client))`, and a recurrence gate (`tests/_guardrails/test_no_cli_client_patch_surface.py`) forbids re-exposing the name on any `*_cmd` module. The eight pure re-export `cli/services/*` shells that survived only to preserve import / call-time seams were collapsed into their `_app/` cores. Note `ctx.obj["client_factory"]` is the CLI **adapter's** client seam — a second front-end (MCP / HTTP) injects its client through the neutral `execute_<verb>(plan, client)` signature, not this Click-specific key.
|
||||
- **Cassette invariance.** The relocation preserves the RPC call set/order/body-shape and the full-id fast paths, so the existing VCR cassettes (matched on `rpcids` + decoded body shape, blind to code path) stay valid **without re-recording**.
|
||||
|
||||
## What stays in the adapter (not relocated)
|
||||
|
||||
The "would a headless server call this?" test decides. Presentation/interactive code stays in `cli/`: Rich rendering, `--json` envelope assembly, exit-code policy, interactive login (playwright / browser-cookie / prompts). Two illustrative calls:
|
||||
|
||||
- `cli/resolve.py` stays a rich **adapter** over the pure `_app/resolve.py` **core** — it adds `ClickException` (not `ValidationError`), console `emit_status`, `entity_name`/`list_command` message hints, the `allow_full_id_passthrough` flag, and a pluggable `error_factory`. It is a justified adapter/core split, not duplication, so it was **not** collapsed.
|
||||
- `agent show` stays pure presentation (no neutral core — nothing a headless caller would invoke).
|
||||
|
||||
## Consequences
|
||||
|
||||
- Additional front-ends reuse the neutral core directly. MCP and REST routes now import `_app.serialize.to_jsonable` / `source_summary` instead of maintaining private serializers.
|
||||
- `_app` gains fast, failure-localizing **unit** tests alongside the CLI's **integration**/cassette tests — layered coverage, not redundancy (measured ≈89 % CLI-over-`_app` line overlap is the unit/integration split, not same-layer duplication).
|
||||
- More indirection (adapter → `_app`), held in check by the boundary lint and the typed contract.
|
||||
- Trade-off accepted: not every CLI test was pushed down; the CLI keeps its end-to-end coverage while `_app` adds direct coverage.
|
||||
|
||||
## Related
|
||||
|
||||
ADR-0008 (CLI-services extraction), ADR-0012 (implementation-surface convention), ADR-0015 (`--json` envelope contract), ADR-0019 (error-and-return contract).
|
||||
@@ -0,0 +1,115 @@
|
||||
# ADR-0022: Regenerable test baselines (derive / store / compare / regen)
|
||||
|
||||
## Status
|
||||
|
||||
Accepted.
|
||||
|
||||
## Context
|
||||
|
||||
Several guardrails freeze a *snapshot* of a value the code already derives, so a
|
||||
change to the public surface is a deliberate, diff-visible act rather than a
|
||||
silent regrowth (the compat audit in `scripts/audit_public_api_compat.py` only
|
||||
flags removed/changed exports vs the last release tag — it is blind to
|
||||
*additions*). Examples:
|
||||
|
||||
- `notebooklm.types.__all__` — the documented, ordered public type surface.
|
||||
- The *collected* public surface (`__all__` + resolvable allowlist extras) of
|
||||
every ungated public module (`notebooklm`, `notebooklm.config`,
|
||||
`notebooklm.exceptions`, …).
|
||||
- The public CLI command tree, options, help, and aliases.
|
||||
|
||||
Historically these were frozen as **hand-typed literals inside the test module**
|
||||
(`_FROZEN_TYPES_ALL`, `_UNGATED_PUBLIC_ALL_SNAPSHOT` in
|
||||
`tests/_guardrails/test_public_surface_manifest.py`). Each was an exact copy of a
|
||||
value the test *already knew how to derive* (`list(notebooklm.types.__all__)`;
|
||||
`_collected_public_surface(module)`). Adding one public symbol therefore meant
|
||||
hand-editing several frozen literals to match values the code re-derives on every
|
||||
run — a high marginal cost and an easy place to introduce a copy error.
|
||||
|
||||
One guardrail already did it the right way: `tests/unit/cli/test_cli_contract.py`
|
||||
*derives* `build_cli_contract()`, commits the result to
|
||||
`tests/fixtures/cli_contract_baseline.json`, asserts `build_cli_contract() ==
|
||||
json.loads(baseline)`, and ships a `__main__` printer to regenerate the file.
|
||||
That pattern — **derive / store / compare / regen** — has the property the inline
|
||||
literals lack: regeneration is one command, and the committed artifact is the
|
||||
reviewed acknowledgement.
|
||||
|
||||
## Decision
|
||||
|
||||
Generalize the CLI-contract pattern into a single **baseline registry** and give
|
||||
it one clean regen command.
|
||||
|
||||
1. **Registry.** `tests/_baselines/registry.py` defines a frozen `Baseline`
|
||||
dataclass `(name, path, derive, sort_keys, …)` and a `BASELINES` list. Each
|
||||
`derive` callable **reuses** the function that already computes the value — it
|
||||
never copies a literal:
|
||||
- `types_all` → `list(notebooklm.types.__all__)` (ordered).
|
||||
- `ungated_surface` → `{module: collect_public_surface(module) …}` over
|
||||
`UNGATED_PUBLIC_MODULES` (ordered lists).
|
||||
- `cli_contract` → `build_cli_contract()` (dict, `sort_keys=True`).
|
||||
|
||||
2. **Committed JSON.** Baselines live under `tests/fixtures/baselines/`
|
||||
(`types_all.json`, `ungated_surface.json`); `cli_contract` keeps its
|
||||
pre-existing path `tests/fixtures/cli_contract_baseline.json`, registered as
|
||||
its `Baseline.path`. Ordered lists serialize in order (lists are never
|
||||
sorted; only dict *keys* are affected by `sort_keys`).
|
||||
|
||||
3. **One freeze test.** `test_baseline_matches_committed_file`
|
||||
(parametrized over `BASELINES`) loads the committed file and asserts it equals
|
||||
`derive()`, and that the committed bytes are in canonical serialized form (so
|
||||
a regen is a no-op on a fresh checkout).
|
||||
|
||||
4. **Regen seam.** A dev-only `--update-baselines` pytest option (in
|
||||
`tests/conftest.py`) flips the freeze test from *assert* to *write
|
||||
`derive()` → `path`*. `scripts/regen_baselines.py` is the discoverable wrapper
|
||||
that shells `pytest … --update-baselines`.
|
||||
|
||||
**Dev-only-regen invariant.** Regeneration only ever happens when a developer
|
||||
passes `--update-baselines`. **CI must NEVER pass the flag — it only diffs.**
|
||||
This is enforced, not merely documented: the `update_baselines` fixture and
|
||||
`scripts/regen_baselines.py` both refuse to regenerate when a CI environment is
|
||||
detected (`CI` env var), failing loudly instead of silently rewriting a baseline.
|
||||
|
||||
## Scope boundary
|
||||
|
||||
- **`_DOCUMENTED_PUBLIC_IMPORTS` stays hand-curated.** It encodes authored
|
||||
*intent* (the promised import surface), not derived reality; regenerating it
|
||||
would make its test tautological.
|
||||
- **`_TOP_LEVEL_TYPE_EXPORTS` stays authored (Phase 2 candidate).** Its naive
|
||||
derivation (names in `notebooklm.__all__` whose object `is` the identical
|
||||
`notebooklm.types` attribute) over-collects — it pulls in the exception and
|
||||
mind-map re-exports that flow through `notebooklm.types` — so it is not a clean
|
||||
derive. Migrating it needs a sharper predicate; deferred.
|
||||
- **Phase 2 (deferred):** the *derived halves* of `tests/fixtures/rpc_golden/*`
|
||||
and the `json_stdout` baselines are only semi-derivable (they need surgical
|
||||
per-field regen) and are out of scope here.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Wanted**
|
||||
|
||||
- **One-command regen.** Adding a public symbol is `python
|
||||
scripts/regen_baselines.py` + reviewing the diff, not editing hand-typed
|
||||
literals.
|
||||
- **No copy drift.** The committed file and the gate derive from the *same*
|
||||
function; they cannot disagree by a typo.
|
||||
- **Same diff-visibility.** A surface addition still produces a reviewed diff
|
||||
line — the deliberate acknowledgement is preserved, just in JSON.
|
||||
|
||||
**Unwanted**
|
||||
|
||||
- **A regen step.** A surface change now requires running the regen command
|
||||
rather than editing inline — but the failing freeze test names the command.
|
||||
- **Indirection.** The frozen value lives in a JSON fixture, not in the test
|
||||
source; the registry is the index that ties name ↔ derive ↔ file.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Keep hand-typed literals.** Rejected: the per-symbol edit cost and copy-error
|
||||
risk are exactly the friction this ADR removes.
|
||||
- **Auto-accept on mismatch (no committed file).** Rejected: it forfeits the
|
||||
diff-visible acknowledgement — a silent surface regrowth is the failure mode
|
||||
these gates exist to prevent.
|
||||
- **One bespoke regen per baseline (status quo of the CLI contract).** Rejected:
|
||||
it does not scale; each new baseline would re-invent the derive/store/compare
|
||||
plumbing and its own regen entry point.
|
||||
@@ -0,0 +1,79 @@
|
||||
# ADR-0023: Master-token headless auth
|
||||
|
||||
## Status
|
||||
|
||||
Accepted.
|
||||
|
||||
## Context
|
||||
|
||||
The client authenticates to consumer NotebookLM with browser-captured Google
|
||||
cookies (`storage_state.json`). Those cookies are short-lived: `__Secure-1PSIDTS`
|
||||
rotates, `__Secure-1PSID` is eventually culled, and there is no unattended way to
|
||||
re-acquire them — a human must re-run `notebooklm login` in a browser. This makes
|
||||
true headless / long-lived / CI usage fragile (see persona D in
|
||||
[installation.md](../installation.md)).
|
||||
|
||||
The #1638 spike evaluated an Android gRPC backend as a cookie-free alternative
|
||||
and, in doing so, proved something smaller and more useful: a durable Google
|
||||
**master token**
|
||||
(`aas_et/…`, obtained once from `accounts.google.com/EmbeddedSetup`) can **mint
|
||||
fresh NotebookLM web cookies on demand** off-device, no browser per session
|
||||
(`OAuthLogin → uberauth → MergeSession`). The minted cookies authorize the entire
|
||||
existing web surface (verified: `batchexecute LIST_NOTEBOOKS` → 200, 61
|
||||
notebooks; `from_storage` lists live).
|
||||
|
||||
Two options followed: **A** — mint cookies from the master token and reuse the
|
||||
existing web client; **B** — build a second Android gRPC backend. B is a large,
|
||||
separately-maintained RPC surface with open problems (upload endpoint, media
|
||||
download). A solves the original headless-auth problem with a small diff.
|
||||
|
||||
## Decision
|
||||
|
||||
Implement **Option A**; defer Option B.
|
||||
|
||||
- A new `[headless]` extra (`gpsoauth`) and `_auth/master_token.py` mint cookies
|
||||
from the master token. `notebooklm login --master-token` bootstraps (one
|
||||
browser sign-in to capture the single-use `oauth_token`, then durable),
|
||||
`--master-token-refresh` re-mints. The token is stored `0600` at
|
||||
`master_token.json` beside the profile's `storage_state.json`.
|
||||
- Minted cookies are written into the normal `storage_state.json`; the existing
|
||||
loader, inline `__Secure-1PSIDTS` recovery, keepalive, and persistence run
|
||||
**unchanged** (the minted jar carries `SID`+`APISID`+`SAPISID`, so recovery
|
||||
mints PSIDTS on first load).
|
||||
- **Recovery (the one library touch-point):** when a `master_token.json` is
|
||||
present, an expired session re-mints in-process as **layer-4** of
|
||||
`refresh_auth_session`, after the existing homepage / `RotateCookies` /
|
||||
headless-browser ladder is exhausted — exactly where the client previously
|
||||
raised "run 'notebooklm login'". It reaches the code via the
|
||||
`AuthRefreshCoordinator` single-flight, so concurrent RPCs coalesce one
|
||||
re-mint. This is the "PSIDTS recovery is different for master-token profiles"
|
||||
decision: re-mint replaces the hard-fail; routine `RotateCookies` keepalive is
|
||||
unchanged.
|
||||
- The CI env-var path (`NOTEBOOKLM_MASTER_TOKEN`) is **deferred** — shipping the
|
||||
`master_token.json` file (like `storage_state.json`) covers CI today, and an
|
||||
inline token would still need cookies written to disk for recovery to work.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Headless auth is solved for the web client** with no per-session browser and
|
||||
automatic recovery — independent of ever building the Android backend.
|
||||
- **Security:** the master token is full-account, durable, and survives password
|
||||
changes until explicitly revoked — a materially larger blast radius than an
|
||||
expiring `storage_state.json`. Mitigations: dedicated/throwaway account only,
|
||||
`0600`, strict redaction (no token/`oauth_token`/`ya29`/cookie in logs or
|
||||
errors; third-party urllib3/requests DEBUG bodies suppressed around gpsoauth),
|
||||
prominent doc warnings. The flow uses Google's unofficial Android auth path
|
||||
(`gpsoauth`) and is ToS-grey like the rest of the client.
|
||||
- **Single-consumer per account:** each re-mint creates a new session, so N
|
||||
concurrent workers re-minting the same account can invalidate each other's
|
||||
`SID`. In-process re-mint is coalesced; cross-process callers should treat one
|
||||
account as single-consumer.
|
||||
- **Risks / open items:** DBSC could one day reject server-minted cookies
|
||||
(re-mint is the mitigation while it isn't enforced); `gpsoauth.exchange_token`
|
||||
is the fragile call (pinned `>=1.1.0`, no `<2` cap so the 2.0.0 `ServiceDisabled`
|
||||
fix installs); master-token durability over weeks is unverified (a durability
|
||||
cron is the follow-up). Cold-dead cookies at process start are recovered by
|
||||
`notebooklm login --master-token-refresh`.
|
||||
- **Option B (a full Android gRPC backend) is deferred** as a DBSC hedge; the
|
||||
master token already solves headless auth for the web client, so building a
|
||||
second RPC surface is not justified now.
|
||||
@@ -0,0 +1,390 @@
|
||||
# ADR-0024: MCP remote file transfer (signed-URL side-channel)
|
||||
|
||||
## Status
|
||||
|
||||
Proposed.
|
||||
|
||||
## Context
|
||||
|
||||
The MCP server now runs over a remote HTTP transport for the claude.ai connector
|
||||
(ADR-precedent: #1645 remote transport, #1647 self-hosted OAuth). Two tools still
|
||||
assume the **stdio** model where the server's filesystem *is* the user's
|
||||
filesystem:
|
||||
|
||||
- `source_add` with `source_type="file"` takes `path` — a path **on the server
|
||||
host** (`src/notebooklm/mcp/tools/sources.py:160`).
|
||||
- `artifact_download` takes `path` — the output file **on the server host**
|
||||
(`src/notebooklm/mcp/tools/artifacts.py:344`).
|
||||
|
||||
Over the claude.ai connector the server is in a container behind a
|
||||
Cloudflare/Tailscale tunnel. A path argument the user supplies refers to a
|
||||
filesystem they cannot see, and a file the server writes lands somewhere they
|
||||
cannot reach. So "upload a local PDF" and "download my podcast" are both broken
|
||||
on the remote connector even though they work over stdio.
|
||||
|
||||
MCP itself offers no help here:
|
||||
|
||||
- **No upload affordance.** claude.ai does not stream a user-selected local file
|
||||
into a tool call. Tool arguments are JSON; base64-in-a-string has no client UI,
|
||||
is not wired to chat-attached files, and dies on request-size limits. (#1803 later
|
||||
added `source_upload_bytes` for the narrow case where an agent already *holds* the
|
||||
bytes and neither the browser nor the `agent_upload` POST can move them: it passes
|
||||
≤10,000 chars of base64 in-channel — ≈7 KB — and the connector adds it server-side.
|
||||
The request-size limit is exactly why that cap is so small; anything larger still
|
||||
takes the signed-URL flow this ADR defines.)
|
||||
- **No usable download affordance.** A tool *result* can carry base64
|
||||
image/audio/`EmbeddedResource`, but a podcast/video is tens of MB; base64'd
|
||||
through JSON-RPC it blows the channel and claude.ai will not materialize it as a
|
||||
saved file.
|
||||
|
||||
So binaries must travel **outside** the MCP JSON-RPC channel.
|
||||
|
||||
The user confirmed local-binary **upload** (e.g. a PDF on their laptop) is in
|
||||
scope, not just URL/YouTube/Drive sources. So both directions need a real
|
||||
byte-transfer path.
|
||||
|
||||
## Prior art & ecosystem alignment (2026)
|
||||
|
||||
Before designing this we checked whether the MCP ecosystem has standardized file
|
||||
transfer (research with citations; key sources below). Findings:
|
||||
|
||||
- **Upload has NO native MCP primitive.** The official File Uploads Working Group
|
||||
charter (2026-04-23, Anthropic) states servers today "resort to prose
|
||||
instructions asking for base64 strings or local paths"; the proposed
|
||||
declarative-file-input descriptor (**SEP-2356**) is **Draft**, not shippable,
|
||||
and the charter lists **presigned upload URLs** as a candidate approach. So the
|
||||
signed-URL side-channel is the **current accepted best practice**, not a
|
||||
reinvention. The industry convention (FutureSearch, Tigris, LibreChat) is
|
||||
exactly: a tool returns a **presigned/HMAC-signed URL** (short TTL, size +
|
||||
content-type enforced, single-use), the bytes move **out-of-band**, and only a
|
||||
lightweight reference rides the protocol — "the context window is for control
|
||||
messages, not bulk data."
|
||||
- **Download HAS a native primitive** — Resources + `BlobResourceContents`, and
|
||||
claude.ai connectors do support binary resources — **but** custom-connector tool
|
||||
results are capped at **~150 000 characters** (~110 KB binary after base64).
|
||||
Every NotebookLM artifact (podcast, video, slide deck, PDF) is far larger, so
|
||||
the native path is unusable for our payloads and a **signed download URL is the
|
||||
right call**. We return it as a `resource_link` so claude.ai renders it clickable.
|
||||
|
||||
How this design aligns:
|
||||
|
||||
- The upload endpoint accepts a **raw body over POST *or* PUT**, so it serves both
|
||||
delivery paths the ecosystem uses: a human opening the link and uploading via
|
||||
the browser (`fetch` POST), **and** Claude's **code-execution sandbox** `curl`-ing
|
||||
a file it already holds to the presigned URL (the FutureSearch pattern). One
|
||||
handler, both work — no extra code.
|
||||
- **Operational constraint (must document):** the sandbox-PUT path only works if
|
||||
the user has **Code Execution enabled and the server domain whitelisted**
|
||||
(claude.ai Settings → Capabilities → additional allowed domains); otherwise the
|
||||
PUT fails despite a valid signature. The browser-upload path has no such
|
||||
requirement and is the universal fallback.
|
||||
- **Migration note (forward path = SEP-2631).** The standardization most likely to
|
||||
land is **SEP-2631 "File Objects and Transfer"** (Draft, opened 2026-04-22; extends
|
||||
SEP-2356) — and it *standardizes this very side-channel*: protocol-native
|
||||
`files/authorizeUpload` / `files/authorizeDownload` control-plane methods that hand
|
||||
the client a presigned **out-of-band** HTTPS URL (bytes stay out of JSON-RPC), plus
|
||||
`x-mcp-file` URI-string inputs and a `FileValue` output type (uri / displayName /
|
||||
mimeType / size / digest). So this ADR is **forward-compatible, not a stopgap to be
|
||||
thrown away**: the existing signed-URL endpoints (`_filelink` / `_fileroutes`) become
|
||||
the presigned targets the client negotiates, and migration mainly moves the UX into
|
||||
the client's **native file picker** — removing the manual browser round-trip that is
|
||||
today's upload friction. Blocked on SEP-2631 landing **and** FastMCP + claude.ai
|
||||
implementing it; until all three, the side-channel is both the shippable approach and
|
||||
the pattern the spec is converging on. (SEP-2356's declarative-file-input alone, being
|
||||
base64-leaning, was never sufficient for our payload sizes.) Tracked in #1656.
|
||||
|
||||
Sources: WG charter https://modelcontextprotocol.io/community/file-uploads/charter
|
||||
· SEP-2356 https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2356
|
||||
· SEP-2631 (File Objects and Transfer) https://github.com/modelcontextprotocol/modelcontextprotocol/pull/2631
|
||||
· claude.ai connector guide https://claude.com/docs/connectors/building
|
||||
· FutureSearch upload pattern https://futuresearch.ai/blog/mcp-large-dataset-upload/
|
||||
· Tigris signed-download https://www.tigrisdata.com/blog/mcp-server-sharing/
|
||||
|
||||
## Decision
|
||||
|
||||
Add a **signed-URL HTTP side-channel mounted on the same FastMCP http app**. The
|
||||
MCP tools broker short-lived signed URLs; the user's browser does the actual byte
|
||||
transfer directly against the tunnel. No bytes cross MCP/claude.ai.
|
||||
|
||||
Two custom routes on the FastMCP app (verified: `FastMCP.custom_route(path,
|
||||
methods=[...])` exists in fastmcp 3.2.0, takes a Starlette `Request -> Response`):
|
||||
|
||||
```
|
||||
GET /files/dl/{token} -> stream the artifact (Starlette FileResponse)
|
||||
GET /files/ul/{token} -> minimal upload page (file picker + fetch POST)
|
||||
POST|PUT /files/ul/{token} -> stream raw body -> add source
|
||||
```
|
||||
(`POST` serves the browser `fetch`; `PUT` serves a sandbox `curl` — same handler.)
|
||||
|
||||
**Download flow** (`artifact_download`, http transport):
|
||||
1. Tool returns the signed link as a `resource_link` content item (claude.ai
|
||||
renders it clickable) plus `{ "status": "download_ready",
|
||||
"url": "<base>/files/dl/<token>", "expires_at": … }` — instead of writing to a
|
||||
server path. (A native `BlobResourceContents` is not used: the ~150 KB connector
|
||||
cap rules it out for real artifacts.)
|
||||
2. Browser GET → handler verifies the token, runs the existing download core into
|
||||
a private temp dir, returns `FileResponse` with a `BackgroundTask` cleanup
|
||||
(mirrors `server/routes/artifacts.py:327`).
|
||||
|
||||
**Upload flow** (`source_add type=file`, http transport):
|
||||
1. Tool returns `{ "status": "upload_required", "url": "<base>/files/ul/<token>",
|
||||
"expires_at": … }` instead of reading a server path.
|
||||
2. Browser GET → one-screen HTML page with a file picker.
|
||||
3. The page uploads the file as a **raw request body** via
|
||||
`fetch(url + "?filename=" + encodeURIComponent(file.name),
|
||||
{method:"POST", headers:{"Content-Type": file.type || "application/octet-stream"},
|
||||
body: file})` — **not** a multipart form. A raw body omits the filename, and
|
||||
NotebookLM's upload **requires** the real basename+extension (an extensionless
|
||||
name 400s, `_source/upload.py:411`), so the page passes the browser-selected
|
||||
`file.name` as a query param and the type as `Content-Type`. The handler reads
|
||||
`request.stream()` chunk-by-chunk into a `0o600` temp file (named from the
|
||||
**sanitized** `?filename`) with a **running byte cap** (the real DoS defense),
|
||||
after an early `Content-Length` reject. It then runs the neutral `source_add`
|
||||
core with `source_type="file"`, deletes the temp in a `finally` (incl.
|
||||
mid-stream disconnect), and returns an HTML "added (id=…)" page.
|
||||
4. The agent confirms with `source_wait` / `source_list`.
|
||||
|
||||
The raw-body upload (vs. `<form enctype=multipart/form-data>` /
|
||||
`request.form()`) is deliberate: it avoids the `python-multipart` dependency
|
||||
(present only in the `server` extra, **not** `mcp`) and avoids Starlette spooling
|
||||
the entire multipart body to disk *before* the per-chunk size check can run — the
|
||||
disk-exhaustion hole that the REST route only closes with app-level
|
||||
Content-Length middleware the FastMCP custom routes do not inherit. The
|
||||
agent-supplied `title` / `mime` are carried in the **upload token** (signed, not
|
||||
user-tamperable); the **filename** (which the token cannot know — it is minted
|
||||
before the user picks a file) arrives as the sanitized `?filename=` query param
|
||||
from the browser/sandbox and seeds the temp file's basename+extension.
|
||||
|
||||
### Stateless, token-encoded — no registry, one scoped inline sweeper (`ul`)
|
||||
|
||||
The signed token **encodes the operation parameters**, so the handlers hold no
|
||||
server-side state:
|
||||
|
||||
- download token payload: `{op:"dl", nb, atype, fmt?, aid?, exp, jti}`
|
||||
- upload token payload: `{op:"ul", nb, title?, mime?, exp, jti}`
|
||||
|
||||
Carrying `title`/`mime` in the upload token preserves the current
|
||||
`source_add type=file` parameters across the browser round-trip — and because
|
||||
they are signed, the uploader cannot tamper with them.
|
||||
|
||||
Token = `base64url(json(payload)) + "." + base64url(HMAC-SHA256(key, body))`
|
||||
(stdlib `hmac`/`hashlib`/`base64`/`json`/`secrets` — no new dependency;
|
||||
`itsdangerous` is not installed). Verify enforces a max token length **before** any
|
||||
decode/HMAC work, re-pads base64url, recomputes the MAC in constant time
|
||||
(`hmac.compare_digest`), and checks `exp`.
|
||||
|
||||
There is **no ref-registry and no *background* sweeper**. The one piece of state
|
||||
is the `ul` single-use tracker added by #1746 (see *Residual risk* below): an
|
||||
ephemeral, bounded, **inline-swept** in-process set of consumed `jti`s that dies
|
||||
with the process just like the signing key. `dl` remains fully stateless
|
||||
(multi-use within its TTL). For downloads the token is still the state.
|
||||
|
||||
### Auth model (verified against fastmcp 3.2.0)
|
||||
|
||||
The browser opening a signed URL **cannot** carry the MCP bearer/OAuth
|
||||
credential, so the side-channel must authenticate itself. This is exactly how
|
||||
FastMCP mounts routes (`fastmcp/server/http.py`):
|
||||
|
||||
- `auth.get_middleware()` is added as **global** middleware — it *authenticates*
|
||||
(populates request scopes) but does **not** reject unauthenticated requests
|
||||
(`BearerAuthBackend.authenticate` returns `None`, it does not raise; `MultiAuth`
|
||||
/ `McpBearerAuthProvider` inherit this non-rejecting base).
|
||||
- `RequireAuthMiddleware` wraps **only** the MCP route (streamable-http transport:
|
||||
`http.py:333-343`).
|
||||
- Custom routes (streamable-http block: `http.py:355-357`) are appended unwrapped
|
||||
and reached **without** the bearer gate.
|
||||
|
||||
Therefore the **HMAC signed token is the sole, sufficient auth** for `/files/*`,
|
||||
which is correct (the browser has no bearer). A regression test pins this
|
||||
FastMCP behavior so an upgrade that starts gating custom routes fails loudly.
|
||||
|
||||
### Signing key and public base URL
|
||||
|
||||
- **Key:** ephemeral `secrets.token_bytes(32)` generated at server start. Tokens
|
||||
are short-TTL (upload **15 min**, download **30 min**); a restart invalidating
|
||||
outstanding links is acceptable and removes a secret to manage. No config. The
|
||||
token rides in the URL **path**, so it is captured by tunnel access logs
|
||||
(Cloudflare/Tailscale), browser history, and any `Referer` — the short TTL +
|
||||
`no-referrer` bound the window, and the blast radius is the single tenant's own
|
||||
account (a one-shot download token would be needed for multi-tenant; the
|
||||
stateless design wins here).
|
||||
The payload is **signed, not encrypted**: a leaked URL exposes the base64-decoded
|
||||
notebook id / artifact type / title to whoever reads the log. Accepted — these are
|
||||
the single tenant's own low-sensitivity metadata, and the HMAC's job is to prevent
|
||||
*forgery*, not disclosure (encrypting would require opaque server-side state,
|
||||
defeating the stateless design). `PUBLIC_URL` rejects userinfo (`user:pass@host`);
|
||||
it intentionally does NOT reject private/loopback hosts (operator-set config that
|
||||
mirrors the OAuth base-URL validator — a private value just yields a link the
|
||||
operator's own browser can't reach, not an exfiltration vector).
|
||||
- **Base URL:** `NOTEBOOKLM_MCP_PUBLIC_URL`, falling back to
|
||||
`NOTEBOOKLM_MCP_OAUTH_BASE_URL` (the public https tunnel URL the OAuth flow
|
||||
already requires). Either value is validated as a **bare https origin** by the
|
||||
same check `_oauth.py` already applies to the OAuth base URL (https scheme, no
|
||||
path/query/fragment) — extracted into a shared `_validate_bare_https_origin`
|
||||
helper so a `/mcp`-suffixed or non-https value can't produce broken/unsafe links.
|
||||
|
||||
**No startup crash.** File transfer is an *optional* capability. A bearer-only
|
||||
remote deployment with no public URL set is still valid (chat/etc. work) — the
|
||||
server does **not** `SystemExit`. The config is simply absent, and the two file
|
||||
tools fail clean **at call time** with "remote file transfer is not configured;
|
||||
set NOTEBOOKLM_MCP_PUBLIC_URL". (An earlier draft proposed a startup `SystemExit`;
|
||||
that would break bearer-only remote servers that never use file transfer — both
|
||||
reviews flagged it, and it is rejected.)
|
||||
|
||||
### Transport branch
|
||||
|
||||
`create_server` gains an optional file-transfer config (signer + validated public
|
||||
base URL), built only in the `__main__.py` http branch and carried on `AppState`.
|
||||
When present the two tools emit URLs; when absent (stdio, or http without a public
|
||||
URL) they keep / fall back to the existing path-based behavior **unchanged** on
|
||||
stdio, or the clean "not configured" error on http. stdio is untouched.
|
||||
|
||||
### Reaching the live client from a custom route
|
||||
|
||||
A `custom_route` handler receives a Starlette `Request`, not an MCP `Context`, so
|
||||
it cannot use the tools' `get_client(ctx)`. The single process-wide client is
|
||||
reached via `request.app.state.fastmcp_server` (FastMCP sets this on the Starlette
|
||||
app) → `._lifespan_result` (the `AppState` yielded by the lifespan), guarded by
|
||||
`._lifespan_result_set`. `_lifespan_result` is a FastMCP private; a regression
|
||||
test pins both this access path and the no-bearer reachability so a FastMCP
|
||||
upgrade that changes either fails loudly. Handlers take `(request)` only and read
|
||||
`request.path_params["token"]` (a `(request, token)` signature crashes Starlette's
|
||||
`request_response`).
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
- Local-binary upload and artifact download work over the claude.ai connector.
|
||||
- Reuses the proven cores: `_app.download` + `download_core.execute_download`, the
|
||||
`source_add` file core + `validate_upload_path`, the 200 MiB cap, the temp-spool
|
||||
+ `BackgroundTask` cleanup patterns from `server/routes/*`.
|
||||
- No new dependency; no persistent state; no background sweeper (the #1746 `ul`
|
||||
single-use tracker is an ephemeral, inline-swept in-process set — no background task).
|
||||
- stdio behavior unchanged.
|
||||
|
||||
**Negative / risks**
|
||||
- Depends on FastMCP not force-gating custom routes (pinned by a regression test).
|
||||
- A leaked upload URL within its 15-min TTL lets someone add a source **to the
|
||||
single tenant's own notebook**; a leaked download URL within 30 min streams one
|
||||
artifact. Single-tenant blast radius, short TTL. **Update (#1746):** `ul` replay-burn
|
||||
is **no longer skipped** — upload tokens are now single-use (one *successful* add per
|
||||
token), because a leaked `ul` link is a content-agnostic write/injection primitive
|
||||
(see *Residual risk* below). `dl` replay-burn **is** still deliberately skipped
|
||||
(multi-use within its 30-min TTL — Range/resume needs it). Note the tension:
|
||||
shortening `UPLOAD_TTL` to bound a leak also shrinks the retry window — a 200 MiB
|
||||
upload over a poor link can take ~13 min, near the 15-min TTL — so `ul` single-use
|
||||
is recorded **on success only**, leaving a failed-then-retried upload able to reuse
|
||||
the same link. Don't shorten `ul` without weighing that.
|
||||
A leaked upload token's *concurrent* replay is now also bounded by the atomic jti
|
||||
claim (`try_begin`): a second concurrent POST of the same token is rejected before it
|
||||
spools, so the pre-#1746 N×200 MiB concurrent-spool window collapses to the narrow
|
||||
race of two POSTs that both pass the claim before either commits — itself capped by
|
||||
`_MAX_CONCURRENT_UPLOADS` and self-cleaning (`finally: _cleanup`).
|
||||
- Two new internet-facing routes on the tunnel — covered by: a streamed byte cap
|
||||
(real defense) plus a `Content-Length` early-reject (413); `title`/`mime` carried
|
||||
in the signed token (not user-tamperable) and the `?filename` basename-sanitized;
|
||||
the reused path-traversal guard +
|
||||
served-path-inside-tempdir assertion; token binding to op+notebook with a
|
||||
pre-decode length cap. The signed-token is itself in the URL path, so the HTML
|
||||
pages send `Referrer-Policy: no-referrer` + `Cache-Control: no-store`, set
|
||||
`X-Frame-Options: DENY` / a restrictive CSP, and HTML-escape all interpolated
|
||||
values; the short TTL bounds the value of a token that leaks via history/logs.
|
||||
- The upload page is a tiny static HTML page (file picker + a `fetch()` POST); not
|
||||
a polished UI.
|
||||
|
||||
**Rejected alternatives**
|
||||
- *Bytes through MCP results (base64).* Size limits + claude.ai won't save files.
|
||||
- *Stateful upload broker with a ref registry + TTL sweep.* Unneeded once the
|
||||
token encodes params; more code and a background task for no benefit.
|
||||
- *Reuse the FastAPI `server/` extra routes directly.* Different deployment
|
||||
(separate ASGI app, FastAPI `Depends`); the side-channel belongs on the MCP app.
|
||||
Shared *logic* is reused; the FastAPI plumbing is not.
|
||||
|
||||
### Residual risk: signed-token replay within TTL (`ul` now single-use — #1746; `dl` accepted)
|
||||
|
||||
`FileLinkSigner.verify` (`_filelink.py`) checks the length cap, the HMAC, `exp`, and
|
||||
the `op` claim; a token that passes those checks passes them **every time** until it
|
||||
expires. Leakage is plausible, not theoretical: the token rides in the URL **path**,
|
||||
so it is captured by claude.ai, browser history, tunnel access logs
|
||||
(Cloudflare/Tailscale), and any `Referer`. The `no-referrer` / `no-store` headers
|
||||
narrow, but do not eliminate, that surface. The two ops differ in blast radius:
|
||||
|
||||
- A leaked **`ul`** token is a **content-agnostic write primitive**: its payload is
|
||||
only `{nb, title?, mime?}` and the uploaded bytes are the raw request body, so
|
||||
whoever holds the link can POST **arbitrary content** (≤200 MiB) as a *source* into
|
||||
the owner's notebook — a content / prompt-injection vector, not merely a same-file
|
||||
replay.
|
||||
- A leaked **`dl`** token can **re-exfiltrate** the current latest artifact of that
|
||||
type until it expires (`DOWNLOAD_TTL` = 30 min). The token pins
|
||||
`{nb, atype, fmt?, aid?}`, not a byte snapshot, so a replay serves whatever the
|
||||
download core resolves at replay time.
|
||||
|
||||
**Decision (updated by #1746): `ul` is single-use; `dl` stays multi-use.**
|
||||
|
||||
`ul` — **single-use enforced.** Every token now carries a random `jti` (128-bit
|
||||
`secrets`, injected by `sign()` and covered by the MAC). The `/files/ul` POST route
|
||||
atomically **claims** the jti (`ConsumedJtiStore.try_begin`) before spooling and
|
||||
**commits** (burns) it only after a *successful* `source_add`; a failed / aborted /
|
||||
429'd upload **rolls back** the claim from the route's `finally` so the link can be
|
||||
retried. Consequences:
|
||||
|
||||
- A sequential replay (the realistic leaked-from-logs case) is rejected with a flat
|
||||
403 before any spool, once the token's one successful use has happened.
|
||||
- The atomic claim also rejects a *concurrent* duplicate POST before it spools,
|
||||
collapsing the pre-#1746 N×200 MiB concurrent-spool window to the narrow race of two
|
||||
POSTs that both pass the claim before either commits (bounded by
|
||||
`_MAX_CONCURRENT_UPLOADS`).
|
||||
- Recording **on success only** preserves the large-file retry window this ADR
|
||||
protects (a 200 MiB upload over a poor link can take ~13 min): a failed upload does
|
||||
not burn the link.
|
||||
|
||||
`dl` — **replay-within-TTL accepted (multi-use).** The `jti` is present but **not**
|
||||
enforced for downloads, because `GET /files/dl/{token}` is streamed directly and a
|
||||
`Range`/resumable client legitimately re-issues the GET (a reconnect with `Range:`)
|
||||
to resume a dropped stream — single-use would 403 the resume and break large-artifact
|
||||
downloads. `dl` is also lower severity (re-reads the *same* artifact; not a write
|
||||
primitive) and is already bounded by:
|
||||
|
||||
- **Short TTL** — 30 min caps the replay window.
|
||||
- **Ephemeral per-process signing key** — `secrets.token_bytes(32)` minted at server
|
||||
start invalidates every outstanding token on restart.
|
||||
- **Single-tenant scope** — blast radius is the operator's own account; no cross-tenant
|
||||
escalation.
|
||||
- **HMAC integrity** — a token cannot be forged or tampered with; replay needs a
|
||||
*legitimately issued* token.
|
||||
- **Download concurrency cap** — `_MAX_CONCURRENT_DOWNLOADS = 4` (#1681) bounds
|
||||
concurrent in-flight downloads and holds each slot for the artifact's whole temp-disk
|
||||
lifetime, released from a `finally` at end-of-stream (a `FileResponse` subclass), so
|
||||
a replay cannot fan out unbounded fetches and a disconnect/aborted `Range` cannot
|
||||
leak a slot.
|
||||
|
||||
**Why the `ul` reversal is consistent with the stateless design.** The two original
|
||||
objections to a `jti` seen-set are answered, not ignored: the store is **ephemeral,
|
||||
bounded (8192), and inline-swept** — it dies with the process exactly like the signing
|
||||
key, and its non-durability across a restart is moot because the key rotation already
|
||||
invalidates every token on restart. There is still **no ref-registry and no background
|
||||
sweeper**; `dl` remains fully stateless.
|
||||
|
||||
**What triggered the reversal, and what would change it further.** The original ADR
|
||||
enumerated three conditions that would flip the tradeoff — multi-tenant, TTLs
|
||||
lengthened, or a reported replay incident. #1746 is **none of those**; it is an added
|
||||
**fourth** trigger: the 2026-07-02 multi-model MCP gap review **re-weighted the `ul`
|
||||
severity** — reframing a leaked upload token as a *write / content-injection
|
||||
primitive* rather than a same-file re-read — which raised it above the "re-exfiltrate
|
||||
low-sensitivity metadata" framing the original decision weighed. The three original
|
||||
conditions still stand as reasons to revisit **`dl`** single-use and/or shorten the
|
||||
TTLs; none has fired for `dl`, and `dl`'s Range/resume constraint is the standing
|
||||
reason it stays multi-use.
|
||||
|
||||
### Why the REST server is out of scope
|
||||
|
||||
The REST server (`server/` extra) **already** supports binary file transfer
|
||||
natively: `POST /v1/notebooks/{id}/sources/file` (multipart upload,
|
||||
`server/routes/sources.py:178`) and `POST /v1/notebooks/{id}/artifacts/download`
|
||||
(`FileResponse`, `server/routes/artifacts.py:327`, documented at
|
||||
`installation.md:414`). It needs **no** signed-URL side-channel because a REST
|
||||
client is a programmatic HTTP client that carries the bearer token and streams
|
||||
bytes directly — it has neither of the two constraints that force this design
|
||||
(the claude.ai connector's browser cannot carry the MCP credential into a tool
|
||||
call, and the JSON-RPC channel cannot carry bytes). This ADR is therefore
|
||||
**MCP-only** by design, not by omission.
|
||||
@@ -0,0 +1,75 @@
|
||||
# ADR-0025: MCP tool granularity — mega-tools vs. discrete verbs
|
||||
|
||||
## Status
|
||||
|
||||
Accepted.
|
||||
|
||||
## Context
|
||||
|
||||
The MCP surface is **35 tools** (`tests/unit/mcp/test_manifest.py`, ceiling 40),
|
||||
above the 5–15/server that current guidance recommends (Anthropic *Writing
|
||||
effective tools for agents*, Sep 2025; GitHub cut Copilot 40→13 for measurable
|
||||
accuracy + latency gains). A tool-interface review flagged that two tools carry
|
||||
the opposite problem — they are **mega-tools** whose real contract lives in
|
||||
runtime validators the JSON schema can't express:
|
||||
|
||||
- `source_add` — 10 params, two modes (single via `source_type`, batch via
|
||||
`urls`), every param optional; three runtime validators enforce which
|
||||
combinations are legal (`src/notebooklm/mcp/tools/sources.py`).
|
||||
- `artifact_generate` — 20 params; per-kind option *applicability* is checked at
|
||||
runtime (the option *values* are already `Literal`s pinned to the core maps).
|
||||
|
||||
The "finish the discrete-verb direction" fix (ADR-0021's transport-neutral
|
||||
philosophy applied to the tool boundary) would split these so the schema states
|
||||
each contract. But splitting **raises the tool count**, which collides with the
|
||||
"fewer tools" evidence — unless paired with **progressive disclosure** (deferred
|
||||
tool loading), which cut schema tokens ~85% *and* raised accuracy in Anthropic's
|
||||
Tool Search Tool.
|
||||
|
||||
The decisive constraint: progressive disclosure is a **client/platform** feature.
|
||||
The MCP spec (2025-06-18) has the server advertise its whole tool list via
|
||||
`tools/list`; there is **no server-forced deferred loading**. So an MCP server
|
||||
cannot guarantee a lean in-context surface for arbitrary clients (Claude
|
||||
Desktop/Code, Cursor, …). Ceiling math (**as of the Tier-1 read-merges, which took the
|
||||
surface from 37 to 35**): splitting `source_add` into `source_add_url` / `_file` /
|
||||
`_text` (keeping the existing batch mode) is **+3 tools = 38**, now within the 40
|
||||
ceiling with a little headroom — so the ceiling no longer blocks *that* split by
|
||||
itself; a full `artifact_generate` per-family split (+several) would still breach 40.
|
||||
(At authoring time the surface was 37, making the `source_add` split land at exactly
|
||||
40 — the Tier-1 merges since freed those two slots.)
|
||||
|
||||
## Decision
|
||||
|
||||
**Do not split the mega-tools now.** Specifically:
|
||||
|
||||
1. **`artifact_generate` stays unified.** Its finite options are already `Literal`
|
||||
enums pinned to the core maps; only per-kind applicability is runtime, and a
|
||||
per-family split would breach the ceiling and duplicate the shared
|
||||
`source_ids` / `language` / `style` params across N tools. Improve it instead
|
||||
via leaner docstrings + per-kind examples (the response-shaping phase).
|
||||
2. **`source_add` split is deferred, not adopted.** It is the stronger candidate
|
||||
(mutually-exclusive params, three runtime validators) but it already batches
|
||||
and would consume all remaining ceiling headroom. Revisit only if (a) a
|
||||
client-supported lean-surface mechanism materializes, or (b) we deliberately
|
||||
raise the ceiling with that split as the justification.
|
||||
3. **No progressive-disclosure implementation.** We cannot force it server-side.
|
||||
We keep descriptions lean (so clients that *do* defer pay less) and leave the
|
||||
option of a config that registers a core tool subset as future work, not a
|
||||
committed deliverable.
|
||||
|
||||
The consistency and response-shaping improvements that do NOT touch tool count
|
||||
(uniform mutation envelope, identifier/naming consistency, list pagination,
|
||||
bounded content reads) proceed independently of this decision.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The surface stays at 35/40 with the two mega-tools intact (Tier-1 read-merges cut it from 37; the mega-tool decision here is unchanged); agents keep learning
|
||||
`source_add` / `artifact_generate` validity partly by failed calls (mitigated by
|
||||
the leaner docstrings + examples).
|
||||
- We avoid a count-inflating refactor we cannot pay for with deferred loading.
|
||||
- If Anthropic/other clients standardize server-hintable deferred loading (several
|
||||
MCP SEPs are in flight), this decision should be revisited — the `source_add`
|
||||
split is the first thing to reconsider, with the ceiling raised as its rationale.
|
||||
- The offline tool-eval harness (schema-token cost + param-count proxy) is the
|
||||
tripwire: if either mega-tool grows, or the surface-wide token cost creeps up,
|
||||
the ratchet fails and forces a fresh look.
|
||||
@@ -0,0 +1,89 @@
|
||||
# ADR-0026: MCP Studio surface — notes + artifacts unified
|
||||
|
||||
## Status
|
||||
|
||||
Accepted.
|
||||
|
||||
## Context
|
||||
|
||||
NotebookLM's UI groups user notes and generated artifacts (audio, video, reports,
|
||||
quizzes, mind maps, …) under one **Studio** panel. The MCP surface split them
|
||||
across two domains: 8 `artifact_*` tools and 4 `note_*` tools (12 tools). This
|
||||
diverged from the product's own mental model, and the split forced agents to know
|
||||
*which* domain a given item lived in before acting — notably note-backed mind
|
||||
maps, which are authored as notes but surface as artifacts.
|
||||
|
||||
Following ADR-0025 (discrete typed verbs, no polymorphic mega-tools) and the
|
||||
#1731 tool-improvement program (35 tools), a review asked whether notes and
|
||||
artifacts should merge into one "Studio" surface — and if so, how far, since a
|
||||
rename alone nets only −1 tool.
|
||||
|
||||
## Decision
|
||||
|
||||
Adopt a **Studio surface** that renames the artifact tools and unifies the
|
||||
cross-cutting *read/list/delete* operations across notes and artifacts, while
|
||||
keeping note **authoring** distinct. Net **35 → 32 tools**.
|
||||
|
||||
1. **Rename** `artifact_*` → `studio_*` (8 tools). Internal `client.artifacts.*`,
|
||||
`ArtifactType`, and the `artifact`/`artifact_type`/`artifact_id` params are
|
||||
unchanged — MCP-adapter-only. The tool module is `tools/studio.py` (renamed from
|
||||
`tools/artifacts.py` once `studio_rename` went cross-type — see point 5 — and the
|
||||
module needed a split anyway); cross-type plumbing lives in
|
||||
`tools/_studio_items.py` and the download registry/helpers shared with
|
||||
`_fileroutes.py` in `tools/_studio_download.py`, both extracted to stay under the
|
||||
ADR-0008 1000-line module cap.
|
||||
|
||||
2. **`note_save`** — an upsert folding `note_create` + `note_update` (−1). Mode is
|
||||
keyed solely on `note`: omitted → create (title+content required); given →
|
||||
update (unresolved ref → `NoteNotFoundError`, never a stray create). Precedent:
|
||||
`share_set_user` is an upsert.
|
||||
|
||||
3. **`studio_list(notebook, item?, kind?, limit?, offset?)`** — merges notes +
|
||||
artifacts into one `items` list with a shared **hyphenated** `type`
|
||||
(`note | audio | video | report | quiz | flashcards | mind-map | infographic |
|
||||
slide-deck | data-table`, plus pass-through `unknown`). Folds `note_list` (−1),
|
||||
including its by-ref single fetch via `item`.
|
||||
|
||||
4. **`studio_delete(notebook, item, confirm)`** — cross-type; folds `note_delete`
|
||||
(−1). Resolves `item` over the merged list and routes by resolved type:
|
||||
`note` → `DELETE_NOTE`, artifact → `DELETE_ARTIFACT` (which clears note-backed
|
||||
mind maps through the note system). Preview is unified as
|
||||
`action:"delete_studio_item"` with `item_id`/`type`/`title`.
|
||||
|
||||
5. **`studio_rename(notebook, item, new_title)`** — cross-type, mirroring
|
||||
`studio_delete`. Resolves `item` over the merged list and routes by resolved
|
||||
type: `note` → the content-preserving `execute_note_rename` (get-then-update,
|
||||
so the body is never dropped), artifact → the artifact rename RPC (which routes
|
||||
note-backed mind maps back through the note system). Returns `item_id`/`type`
|
||||
(was `artifact_id`) plus `new_title`/`is_mind_map`. An absent *full UUID* takes
|
||||
the same idempotent carve-out as `studio_delete` (route to the artifact path,
|
||||
which probes `mind_maps.list`), so rename-by-id of a note-backed mind map that
|
||||
is not in the merged list still works.
|
||||
|
||||
Earlier this stayed artifact-scoped on the reasoning that folding note rename
|
||||
would route through a title-only `execute_note_save(content=None)` with no
|
||||
server-confirmed "None = leave unchanged" guarantee. Using the
|
||||
content-preserving `execute_note_rename` core removes that risk, so the tool now
|
||||
spans notes for surface symmetry with `studio_list`/`studio_delete`.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **Breaking** (experimental MCP surface): `note_create`/`note_update`/`note_list`/
|
||||
`note_delete` and `artifact_*` tool names are removed. Wire keys change:
|
||||
`studio_list` returns `items` (was `artifacts`/`notes`); `studio_delete` returns
|
||||
`item_id`/`type` (was `artifact_id`). Notes domain collapses to `note_save`.
|
||||
- **No dedup pass is needed:** `client.notes.list()` drops `NoteRowKind.MIND_MAP`
|
||||
rows, so notes ∩ artifacts = ∅; the merge keeps a defensive dedup-by-id that
|
||||
never fires in practice.
|
||||
- **Known inefficiency:** `studio_list` (and the `studio_delete` resolve step)
|
||||
issue `GET_NOTES_AND_MIND_MAPS` (`cFji9`) **twice** — once via `notes.list`, once
|
||||
via `artifacts.list`'s mind-map facade — plus `gArtLc` once. A single-fetch
|
||||
version needs an `_app` seam below both client APIs and is deferred as
|
||||
out-of-(MCP-adapter-only)-scope.
|
||||
- **`studio_delete` stays idempotent-on-missing:** an absent *full UUID* routes to
|
||||
the artifact delete path (no raise), preserving delete-by-id idempotency; a
|
||||
non-UUID (prefix/title) miss is a real NOT_FOUND. Correct because a *present*
|
||||
note is always found in the merged list and routed to `DELETE_NOTE`.
|
||||
- **Integration coverage:** cross-type `studio_delete` routing and the sharing
|
||||
domain are unit-covered but lack `mcp_vcr` cassettes (recording needs live auth);
|
||||
tracked in #1732 / #1733.
|
||||
@@ -0,0 +1,68 @@
|
||||
# Architecture Decision Records
|
||||
|
||||
This directory holds the canonical decisions that shape the `notebooklm-py` codebase. Each record explains _why_ a load-bearing pattern exists so that future contributors don't re-litigate (or silently re-introduce) the trade-off.
|
||||
|
||||
## How to use this directory
|
||||
|
||||
- Read the relevant ADR before changing the pattern it describes. If you disagree, write a new ADR that supersedes it — do not edit the original past correcting typos.
|
||||
- Numbering is append-only. Retired ADR numbers are never re-used.
|
||||
- Filenames follow `NNNN-short-title.md` (lowercase, kebab-case).
|
||||
- Status values: `Proposed`, `Proposed — <short explanation>`, `Accepted`, `Accepted (retroactive)`, `Accepted (#PR)`, `Accepted (Sunset = <event>)`, `Superseded — <short explanation>`, `Superseded by ADR-NNNN (#PR)`, `Deprecated`, `Rejected`.
|
||||
- Format: lightweight hybrid — six sections in this exact order: _Title heading_ (`# ADR-NNNN: <Title>`), _Status_, _Context_, _Decision_, _Consequences_, _Alternatives considered_. See [0000-template.md](0000-template.md).
|
||||
|
||||
## When an ADR is required
|
||||
|
||||
The pull-request template asks contributors to confirm that any change to the _architectural shape_ of the codebase carries an ADR addition or update. "Architectural shape" means any of:
|
||||
|
||||
- New / removed / relocated modules in `src/notebooklm/_runtime/`, `src/notebooklm/_middleware/`, `src/notebooklm/auth.py`, `src/notebooklm/_auth/`, `src/notebooklm/cli/services/`.
|
||||
- Changes to the contracts between layers (CLI ↔ Client ↔ Core ↔ RPC).
|
||||
- New or retired test patterns (fixtures, monkeypatch policy, conformance tests).
|
||||
- New cross-cutting policies (retry, idempotency, scrubbing, loop affinity).
|
||||
|
||||
Pure bug fixes, additive RPC method IDs, and CLI ergonomics changes do not require an ADR.
|
||||
|
||||
### Status Format Legend
|
||||
|
||||
The ADR Index table utilizes five eras of Status notation to reflect the lifecycle of decisions:
|
||||
|
||||
1. **`Accepted` / `Accepted (retroactive)`** — The decision is currently active and adopted.
|
||||
2. **`Accepted (Tier X PR Y.Z)`** — Historical PR-naming convention from the early refactoring tiers (Tiers 11-12).
|
||||
3. **`Superseded — <note>` / `Superseded by ADR-NNNN (#PR)`** — Canonical supersession forms, either with a short inline explanation or a replacing ADR and PR.
|
||||
4. **`Accepted (#PR)` / `Accepted; <note>` / `Proposed — <note>`** — Short explanatory forms used when a status needs one compact qualifier.
|
||||
5. **`Superseded by <named-PR> (D1/D2 PR-X)`** — Pre-canonical historical supersession form, linking to specific branch-refactoring PRs.
|
||||
|
||||
## Index
|
||||
|
||||
| ADR | Title | Status |
|
||||
| ------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
|
||||
| [0001](0001-layered-core-seams-and-property-bridge-policy.md) | Layered `_core` seams and the property-bridge policy | Superseded — bridge policy retired in session-shrink arc |
|
||||
| [0002](0002-capability-protocol-pattern.md) | Capability Protocol pattern (`SessionCapabilities` fat union) | Superseded by [arch-d2-cutover](https://github.com/teng-lin/notebooklm-py/pull/835) (#835) |
|
||||
| [0003](0003-auth-facade-write-through.md) | `auth.py` write-through facade (`_AuthFacadeModule`) | Superseded — closed by [ADR-0014](0014-feature-local-runtime-adapters.md) (session-decoupling Waves 3a + 4 T2.2 + 5) |
|
||||
| [0004](0004-loop-affinity-contract.md) | Loop-affinity contract for `NotebookLMClient` | Accepted (retroactive) |
|
||||
| [0005](0005-idempotency-taxonomy.md) | Mutating-RPC idempotency taxonomy | Accepted (retroactive) |
|
||||
| [0006](0006-vcr-scrubber-strategy.md) | VCR cassette scrubber strategy | Accepted (retroactive) |
|
||||
| [0007](0007-test-monkeypatch-policy.md) | Test monkeypatch policy | Accepted |
|
||||
| [0008](0008-cli-services-extraction-pattern.md) | `cli/services/` extraction pattern | Accepted (retroactive) |
|
||||
| [0009](0009-middleware-chain.md) | Middleware chain for cross-cutting transport concerns | Accepted (Tier 12 PR 12.1); context refined by [ADR-0013](0013-composable-session-capabilities.md) (#866) |
|
||||
| [0010](0010-session-kernel-split.md) | Session/Kernel split | Superseded by [ADR-0013](0013-composable-session-capabilities.md) (#866) |
|
||||
| [0011](0011-schema-validation-policy.md) | Schema validation policy (strict-decode default) | Accepted (Tier 13 PR 13.9a) |
|
||||
| [0012](0012-implementation-surface-convention.md) | Implementation surface convention (underscore-prefix policy) | Accepted (Tier 13 PR 13.9a) |
|
||||
| [0013](0013-composable-session-capabilities.md) | Composable Session Capabilities and Feature-Local Runtimes | Accepted |
|
||||
| [0014](0014-feature-local-runtime-adapters.md) | Feature-local runtime adapters as Protocol satisfiers | Accepted (#1082) |
|
||||
| [0015](0015-json-envelope-contract-for-post-parse-click-exceptions.md) | Typed JSON error envelope covers post-parse `ClickException` failures | Accepted |
|
||||
| [0016](0016-auth-identity-and-core-logger-compatibility.md) | Auth identity and core logger compatibility | Accepted |
|
||||
| [0017](0017-public-facade-private-implementation.md) | Public-facade / private-implementation re-export convention | Accepted (retroactive) |
|
||||
| [0018](0018-deprecation-strategy.md) | Deprecation strategy (`_deprecation.py`) | Accepted (retroactive) |
|
||||
| [0019](0019-error-and-return-contract.md) | Error-and-return contract for the public API | Accepted; v0.8.0 implementation landed |
|
||||
| [0020](0020-sealed-async-result-types.md) | Sealed async result types for artifact generation | Proposed — design of record; recommends continued deferral (ADR-0019 Tier 3 / #1345) |
|
||||
| [0021](0021-transport-neutral-app-layer.md) | Transport-neutral application layer (`_app/`) | Accepted |
|
||||
| [0022](0022-regenerable-baselines.md) | Regenerable test baselines (derive / store / compare / regen) | Accepted |
|
||||
| [0023](0023-master-token-headless-auth.md) | Master-token headless auth (Option A) | Accepted |
|
||||
|
||||
ADR-0007 ships alongside its enforcement substrate: the concrete fixtures (`tests/_fixtures/`) and meta-lint (`tests/_guardrails/test_no_forbidden_monkeypatches.py`) are added in the same PR (`arch-d1-fixtures-scaffolding`) so the record is grounded in working code rather than an empty placeholder.
|
||||
|
||||
## Related references
|
||||
|
||||
- [Architecture](../architecture.md) — Canonical overview of the layered architecture.
|
||||
- `docs/development.md` — contributor-facing process notes (testing, releasing, environment setup).
|
||||
- `CLAUDE.md` — onboarding map for AI assistants. Architectural rationale belongs here in `docs/adr/`, not in `CLAUDE.md`.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,919 @@
|
||||
# Auth cookie lifecycle — design notes and field findings
|
||||
|
||||
**Last Updated:** 2026-07-04
|
||||
|
||||
> **Status:** current design notes for the auth refresh stack in `main`.
|
||||
> The numbered recovery ladder below is the canonical taxonomy, shared with
|
||||
> [docs/troubleshooting.md](troubleshooting.md#authentication-errors):
|
||||
>
|
||||
> - **L1** — per-call `RotateCookies` POST (default ON)
|
||||
> - **L2** — periodic background keepalive (`keepalive=N` client kwarg)
|
||||
> - **L3** — headless re-auth from a persisted browser profile / loopback CDP
|
||||
> - **L4** — **master-token re-mint** (`[headless]` extra; no browser, fully automatic)
|
||||
> - **L5** — `NOTEBOOKLM_REFRESH_CMD` external recovery script
|
||||
> - **L6** — manual `notebooklm login`
|
||||
> - **L7** — OS-scheduled `notebooklm auth refresh` (cron / launchd / systemd)
|
||||
>
|
||||
> For long-lived, unattended, headless, server, or CI use, **L4 master-token
|
||||
> re-mint is the recommended path** — see [Recommended setup](#recommended-setup).
|
||||
> Deep field notes, ruled-out experiments, and the internal-persistence hazard
|
||||
> log have been moved to the [Appendix](#appendix-field-notes--historical-findings)
|
||||
> so the body stays focused on durable mechanics.
|
||||
|
||||
## TL;DR
|
||||
|
||||
NotebookLM has no public OAuth surface. The library authenticates by carrying
|
||||
Google session cookies (`SID`, `__Secure-1PSID`, `__Secure-1PSIDTS`, `OSID`, and
|
||||
friends) extracted from a real browser sign-in. Two clocks govern their validity:
|
||||
|
||||
- **`__Secure-1PSIDTS` has a *recommended* rotation cadence of ~600 s**
|
||||
(self-reported by Google as `["identity.hfcr",600]` on the `RotateCookies`
|
||||
response). This is a *hint*, not a hard rejection TTL: the prior value keeps
|
||||
authenticating far longer — commonly hours to days on a stable IP / non-Workspace
|
||||
account. Worst-case profiles (datacenter egress, cross-IP, Workspace policy,
|
||||
incomplete extraction) can collapse that to hours or less.
|
||||
- **`SID` and `__Secure-1PSID`** have very long server-side lifetimes (months to
|
||||
years) and effectively don't expire under normal usage.
|
||||
- **Cookie set completeness matters more than freshness.** Google rejects cookie
|
||||
sets missing `__Secure-1PSIDTS` together with any one other cookie, even though
|
||||
removing `__Secure-1PSIDTS` alone is recoverable — see
|
||||
[§3.3](#33-empirical-cookie-requirements).
|
||||
|
||||
A long-lived client must therefore drive `*PSIDTS` rotation itself. The cleanest
|
||||
mechanism is a direct `POST` to `https://accounts.google.com/RotateCookies` —
|
||||
Google's dedicated unsigned rotation endpoint, the **L1** primitive at the bottom
|
||||
of a tiered recovery design that escalates as failure modes get harder.
|
||||
|
||||
The recovery ladder runs cheapest-to-heaviest — **L1** per-call `RotateCookies`
|
||||
POST, **L2** background keepalive, **L3** headless re-auth / loopback CDP, **L4**
|
||||
master-token re-mint, **L5** `NOTEBOOKLM_REFRESH_CMD`, **L6** manual `notebooklm
|
||||
login`, **L7** scheduled `notebooklm auth refresh` (the same taxonomy as
|
||||
[troubleshooting.md](troubleshooting.md#authentication-errors); per-layer detail in
|
||||
[§4](#4--the-recovery-ladder)). L1/L2 keep a live session fresh but can't revive a
|
||||
dead one; L3 needs the profile's browser session still alive; **L4 is the only
|
||||
fully-automatic layer that revives a fully-expired session with no browser.**
|
||||
|
||||
**L4 (master-token re-mint) is the standout for headless/unattended use.** Unlike
|
||||
L3 it needs no browser at refresh time, and unlike L5/L6 it is fully automatic.
|
||||
One durable master token — one human sign-in, then good for months — re-mints web
|
||||
cookies on demand and self-heals an expired session in-process, coalesced through
|
||||
the `AuthRefreshCoordinator` single-flight. See [§4.4](#44-l4--master-token-re-mint)
|
||||
and [ADR-0023](adr/0023-master-token-headless-auth.md).
|
||||
|
||||
`NOTEBOOKLM_REFRESH_CMD` ([#336](https://github.com/teng-lin/notebooklm-py/pull/336))
|
||||
is a complementary, reactive hook: it runs a user-supplied recovery command on
|
||||
auth-expiry signals, then retries the token fetch once. It is **orthogonal to
|
||||
L1–L4** — those proactively keep the session fresh or re-mint it in-process, while
|
||||
`NOTEBOOKLM_REFRESH_CMD` is the "we lost the session anyway, run my recovery
|
||||
script" lever. See [§6.2](#62-notebooklm_refresh_cmdcommand-line-l5).
|
||||
|
||||
L1 works today on every account type tested. Long-running Python workers should
|
||||
add L2; unattended/headless/server/CI deployments should adopt L4; idle profiles
|
||||
between processes can add L7. If Google extends DBSC enforcement to non-Chrome
|
||||
cookie paths, L3's CDP arm becomes the primary browser-backed recovery path.
|
||||
|
||||
---
|
||||
|
||||
## Available auth methods
|
||||
|
||||
There are five ways to give the library credentials. Pick by deployment shape;
|
||||
they compose (e.g. `--master-token` for the durable credential plus
|
||||
`NOTEBOOKLM_REFRESH_CMD` as a belt-and-suspenders reactive hook).
|
||||
|
||||
| Method | Command / env | Best for | Survives cookie expiry unattended? | Setup cost |
|
||||
|---|---|---|:-:|---|
|
||||
| **(a) Interactive login** | `notebooklm login` (Playwright Google sign-in into a private Chromium profile) | Desktop / interactive use | No — re-login when prompted | Low (one browser sign-in) |
|
||||
| **(b) Browser-cookie reuse** | `notebooklm login --browser-cookies <browser>` (rookiepy extraction from an existing profile) | Reusing a browser you already sign into | Only while the source browser session stays alive; pairs with L7 cron | Low (no interaction) |
|
||||
| **(c) Master token** ⭐ | `notebooklm login --master-token` (`[headless]` extra; durable token, headless L4 re-mint) | **Servers / CI / unattended / headless** | **Yes** — re-mints automatically, no browser | Medium (one bootstrap sign-in, ship `master_token.json`) |
|
||||
| **(d) Inline auth JSON** | `NOTEBOOKLM_AUTH_JSON=<storage_state payload>` | CI / ephemeral containers with no on-disk profile | No — env-var auth has no writeable file, so L3/L4 decline | Low (paste a secret) |
|
||||
| **(e) External refresh hook** | `NOTEBOOKLM_REFRESH_CMD=<command>` | Custom recovery (CookieCloud pull, browser re-extract) layered on any of the above | Depends on the script | Medium (write + secure a script) |
|
||||
|
||||
Notes: **(c) is the recommended default for long-lived headless use** — the only
|
||||
method that both survives full cookie expiry *and* needs no browser at refresh
|
||||
time. **(d) `NOTEBOOKLM_AUTH_JSON`** carries a full `storage_state.json` payload
|
||||
inline (credential-equivalent) with no backing file, so the file-backed recovery
|
||||
layers (L3/L4) decline — good for short-lived CI jobs, but pair with (c) or (e) for
|
||||
anything long-running. **(e) `NOTEBOOKLM_REFRESH_CMD`** is not a credential source
|
||||
on its own; it is the reactive hook that fires after auth has already expired
|
||||
([§6.2](#62-notebooklm_refresh_cmdcommand-line-l5)).
|
||||
|
||||
---
|
||||
|
||||
## Recommended setup
|
||||
|
||||
### Interactive desktop user
|
||||
|
||||
Just `notebooklm login`. The Playwright Chromium flow handles it; re-login when
|
||||
prompted (typically days to weeks between prompts).
|
||||
|
||||
### Long-lived in-process client (agent, MCP server, worker)
|
||||
|
||||
```python
|
||||
async with NotebookLMClient.from_storage(keepalive=600) as client:
|
||||
...
|
||||
```
|
||||
|
||||
L1 fires on `from_storage()`; L2 fires every 600 s while the client is open. This
|
||||
keeps `*PSIDTS` rotating for as long as the process lives.
|
||||
|
||||
### Unattended / headless / server / CI — use the master token (L4)
|
||||
|
||||
This is the recommended path. No browser at refresh time, survives full cookie
|
||||
expiry, and no `storage_state.json` to keep re-shipping.
|
||||
|
||||
1. `pip install "notebooklm-py[headless]"`.
|
||||
2. One-time, on a machine with a browser (dedicated / throwaway account):
|
||||
`notebooklm -p <profile> login --master-token --account you@gmail.com`.
|
||||
3. Ship the bootstrapped profile — **both** `master_token.json` and the
|
||||
`storage_state.json` the bootstrap just minted (each `0600`) — to the server.
|
||||
(A clean server with *only* `master_token.json` and no `storage_state.json`
|
||||
needs one `notebooklm -p <profile> login --master-token-refresh` to mint the
|
||||
initial cookies first; shipping both skips that step.)
|
||||
4. Run commands normally. Cookies are minted on bootstrap and **re-minted
|
||||
automatically** when the session dies (L4, [§4.4](#44-l4--master-token-re-mint));
|
||||
force one by hand with `notebooklm -p <profile> login --master-token-refresh`.
|
||||
|
||||
Caveats: the master token is a full-account, infostealer-grade credential — use a
|
||||
dedicated account, keep the file `0600`, never log or commit it. One account is
|
||||
**single-consumer**: N workers re-minting concurrently can invalidate each other's
|
||||
`SID`. The master token inherits the standing DBSC risk (server-minted cookies
|
||||
could be rejected if enforcement extends to this path), but re-mint is itself the
|
||||
mitigation while it isn't enforced. See
|
||||
[ADR-0023](adr/0023-master-token-headless-auth.md) and
|
||||
[installation.md#d-headless-server-or-ci](installation.md#d-headless-server-or-ci).
|
||||
|
||||
### Unattended without the `[headless]` extra — browser-cookie extract + cron
|
||||
|
||||
If you can't or won't use a master token, extract from a real browser and refresh
|
||||
on a schedule:
|
||||
|
||||
1. Sign in to NotebookLM once in Firefox (or any rookiepy-supported browser).
|
||||
2. `notebooklm -p <profile> login --browser-cookies firefox`.
|
||||
3. Schedule L7: `7,27,47 */1 * * * notebooklm --profile <profile> auth refresh`
|
||||
(off-minute schedule avoids fleet collision).
|
||||
4. Keeping the source browser running with a Google tab adds resilience, but even
|
||||
a closed browser works for hours-to-days while `RotateCookies` keeps
|
||||
succeeding from `SID` alone.
|
||||
|
||||
> **Browser support:** `--browser-cookies` accepts any of the ~16 browsers rookiepy
|
||||
> reads on the host (`arc`, `brave`, `chrome`, `edge`, `firefox`, `opera`, `safari`,
|
||||
> `vivaldi`, …; see `_ROOKIEPY_BROWSER_ALIASES` in
|
||||
> `cli/services/login/cookie_jar.py`). **Firefox is the recommended path on
|
||||
> Windows** because Chrome 127+ App-Bound Encryption makes Chrome reads
|
||||
> admin-or-bust. Scope a Firefox Multi-Account Container with
|
||||
> `firefox::<container-name>` (unscoped extraction merges every container and can
|
||||
> pick the wrong session); scope a Chromium profile with `chrome::<profile>`.
|
||||
|
||||
### Workspace / Enterprise with admin session-binding
|
||||
|
||||
Currently **not supported.** Admin-policy session binding is a Workspace beta that
|
||||
requires DBSC-compatible flows. Request an exemption from your admin or use a
|
||||
personal Google account for automation.
|
||||
|
||||
---
|
||||
|
||||
## 1 · Problem statement
|
||||
|
||||
NotebookLM uses Google's internal `batchexecute` RPC. There is no documented API
|
||||
key, no OAuth scope, no service account path. Every project that automates
|
||||
NotebookLM does so with **scraped session cookies** from a logged-in browser. The
|
||||
library exposes those via `notebooklm login` (Playwright-driven Google sign-in
|
||||
into a private Chromium profile) and `notebooklm login --browser-cookies <browser>`
|
||||
(rookiepy-driven extraction from an existing profile). Both produce a
|
||||
`storage_state.json` that authenticates every subsequent RPC.
|
||||
|
||||
The keepalive question is: **what keeps `storage_state.json` valid between
|
||||
user-driven re-authentications?** The naïve "cookies have expiry timestamps; trust
|
||||
them" answer is wrong on two counts: the most consequential cookie
|
||||
(`__Secure-1PSIDTS`) has a server-side recommended rotation cadence not encoded in
|
||||
its `Expires` attribute (the on-disk `Expires` is irrelevant to server-side
|
||||
validity), and even cookies with a year-long `Expires` are **revoked early by
|
||||
Google's risk model** when the access pattern looks unusual (no JS, no fingerprint,
|
||||
IP changes, long idle gaps). So the library must actively refresh.
|
||||
|
||||
---
|
||||
|
||||
## 2 · Background: Google session auth, rotation, and DBSC
|
||||
|
||||
Vocabulary the rest of the doc uses. Skip to [§3](#3--threat-model) if you've
|
||||
already spent time inside Google's identity surface.
|
||||
|
||||
### 2.1 The cookie taxonomy
|
||||
|
||||
Google authenticates a browser session with a **family of ~15 cookies**, not a
|
||||
single bearer token. Each cookie has a distinct role; the family is designed so
|
||||
revoking or rotating any one slot doesn't invalidate the others. The set is shared
|
||||
across `*.google.com` properties — Search, Drive, Gmail, NotebookLM, YouTube,
|
||||
Workspace — which is why a sign-in to any one produces auth artifacts the rest of
|
||||
the ecosystem accepts.
|
||||
|
||||
Naming conventions:
|
||||
|
||||
- **`__Secure-` prefix.** The cookie's `Secure` attribute must be set, so it's
|
||||
never sent over plaintext HTTP. Google sets this on every meaningful auth cookie.
|
||||
- **`__Host-` prefix.** Stricter: the cookie must also set `Path=/`, must not set
|
||||
`Domain=` (pinned to the exact issuing origin), and must be `Secure`. Used for
|
||||
the most scope-sensitive cookies (`__Host-GAPS`, `__Host-1PLSID`, …).
|
||||
- **`1P` vs `3P`.** First-party vs third-party context. `__Secure-1PSID` is used
|
||||
when the request originates from a `*.google.com` page; `__Secure-3PSID` is the
|
||||
variant Google sends on third-party pages that embed Google content. They rotate
|
||||
independently. We typically need both because intermediate rotation redirects
|
||||
cross the 1P/3P boundary.
|
||||
- **`*SID` / `*SIDTS` / `*SIDCC`.** Three cookie *families* that separate
|
||||
**identity** (who you are, slow to change) from **freshness** (you're using the
|
||||
session right now, fast to expire):
|
||||
|
||||
| Family | Role | Recommended rotation cadence | Stale-value validity |
|
||||
|---|---|---|---|
|
||||
| `*SID` (`SID`, `HSID`, `SSID`, `APISID`, `SAPISID`, …) | Long-lived identity | Months → ~1 year | Practically never expires for active accounts |
|
||||
| `*SIDTS` (`__Secure-1PSIDTS`, `__Secure-3PSIDTS`) | Rotating freshness partner of `*SID` | **~600 s** (Google's self-report) | Hours-to-days on a stable IP / non-Workspace profile |
|
||||
| `*SIDCC` (`SIDCC`, `__Secure-1PSIDCC`, …) | Per-request "session continuity check" | Issued on every request | Not enforced for accept/reject |
|
||||
|
||||
A few cookies sit outside this taxonomy:
|
||||
|
||||
- **`OSID`, `__Secure-OSID`** — per-product session, set on `notebooklm.google.com`
|
||||
and `myaccount.google.com`. Re-issued on each sign-in.
|
||||
- **`LSID`, `__Host-1PLSID`, `__Host-3PLSID`** — identity-service cookies on
|
||||
`accounts.google.com`. Long-lived.
|
||||
- **`__Host-GAPS`** — anti-takeover binding cookie. Long-lived; part of how Google
|
||||
detects suspicious cross-device reuse.
|
||||
|
||||
The library treats these uniformly: extract the full set at sign-in, persist them
|
||||
in `storage_state.json`, replay them on every RPC.
|
||||
`_is_allowed_cookie_domain` (in `_auth/cookie_policy.py`, re-exported through
|
||||
`auth.py`) gates which `Set-Cookie` headers from a redirect chain are worth
|
||||
keeping, matching against `ALLOWED_COOKIE_DOMAINS` plus the regional
|
||||
`google.<cctld>` set.
|
||||
|
||||
### 2.2 How cookie rotation works
|
||||
|
||||
"Rotation" means: the server periodically issues a new value for a short-lived
|
||||
cookie (`Set-Cookie: __Secure-1PSIDTS=<fresh>; …`), and the browser is expected to
|
||||
overwrite its on-disk copy. If the client falls behind, the server eventually
|
||||
stops accepting the old value and the session is dead until re-auth.
|
||||
|
||||
Two clocks run in parallel:
|
||||
|
||||
- The **identity clock** (`*SID`) ticks in months. Google extends it silently as
|
||||
long as it sees activity; for a daily-active user it effectively never expires.
|
||||
- The **freshness clock** (`*PSIDTS`) has a recommended rotation cadence of ~600 s,
|
||||
self-reported in the `RotateCookies` response body as `["identity.hfcr",600]`
|
||||
(`hfcr` = "high-frequency cookie rotation"; `600` = seconds). This is a rotation
|
||||
*hint*, not a hard expiration: stale values keep working for hours or days
|
||||
depending on server-side state, but long-idle sessions eventually drift into
|
||||
sign-in redirects.
|
||||
|
||||
Rotation is **server-driven**: the client posts to a rotation endpoint; the server
|
||||
inspects the existing `*SID` (and optionally a DBSC proof — see §2.3) and returns a
|
||||
fresh `*PSIDTS`. The client only chooses *when* to fire.
|
||||
|
||||
**Crucially: pure RPC traffic against `notebooklm.google.com` does not trigger
|
||||
rotation.** `batchexecute` accepts the existing cookies, but Google only mints a
|
||||
fresh `*PSIDTS` when something talks to the *identity* surface (`accounts.google.com`,
|
||||
the NotebookLM homepage GET, the `RotateCookies` POST). A client that only calls
|
||||
`batchexecute` silently drifts past the rotation window and starts failing — exactly
|
||||
what L1/L2 target. We use `RotateCookies` because it rotates deterministically for
|
||||
both browser-bound and Firefox-extracted sessions (see
|
||||
[Appendix](#appendix-field-notes--historical-findings)).
|
||||
|
||||
### 2.3 Device-Bound Session Credentials (DBSC)
|
||||
|
||||
DBSC is Google's response to **infostealer cookie theft**: malware exfiltrates the
|
||||
cookie jar and an attacker replays it from another machine. DBSC binds a session to
|
||||
**a private key in tamper-resistant hardware** (TPM / Secure Enclave / Strongbox) on
|
||||
the original device — the browser generates a non-extractable keypair at sign-in and
|
||||
registers the public key, then signs a server nonce on every rotation. The enforcing
|
||||
endpoint is **`accounts.google.com/RotateBoundCookies`**, the bound-cookie analog of
|
||||
the unsigned `RotateCookies` we use; an attacker who steals the jar can't sign the
|
||||
next bound rotation, so the stolen session dies instead of renewing. The
|
||||
[W3C spec](https://w3c.github.io/webappsec-dbsc/) is deliberately structured so only
|
||||
hardware-attesting browsers can implement it — no Python HTTP client can, and no
|
||||
public OSS DBSC client exists outside Chrome (see [A3](#a3--ruled-out-experiments)).
|
||||
|
||||
**Current enforcement state.** DBSC is rolling out. Enforcement currently targets
|
||||
**Chrome itself** — Chrome refuses to use cookies that weren't bound at sign-in,
|
||||
even on the same machine. Non-Chrome HTTP clients (httpx, curl, Firefox) can still
|
||||
hit the legacy unsigned `RotateCookies` endpoint without a DBSC proof, so every
|
||||
HTTP-only strategy in this document works today. The day Google extends enforcement
|
||||
to that endpoint, they break together; the in-tree escape is to parasitize a real
|
||||
DBSC-enrolled Chrome session through the L3 CDP attach arm, or to source cookies
|
||||
via an operator-provided `NOTEBOOKLM_REFRESH_CMD` (e.g. CookieCloud federation).
|
||||
See [§7 canaries](#7--canaries-and-signals) for the tripwires that would signal the
|
||||
transition.
|
||||
|
||||
### 2.4 How browser-cookie extraction works
|
||||
|
||||
`notebooklm login --browser-cookies <browser>` reads cookies directly out of an
|
||||
installed browser's profile rather than minting fresh ones via Playwright. It is a
|
||||
**variant of manual login (L6)** and a common backing command for
|
||||
`NOTEBOOKLM_REFRESH_CMD` (L5) — it is **not** a recovery layer of its own, and in
|
||||
particular it is not L4 (L4 is the master token).
|
||||
|
||||
Browsers store cookies in encrypted SQLite databases, with the decryption key in the
|
||||
OS credential store (Keychain / DPAPI / libsecret). **Chrome 127+ adds App-Bound
|
||||
Encryption (ABE)** — a second layer bound to Chrome's signed binary that defeats
|
||||
user-space readers; `browser_cookie3` doesn't handle it and `rookiepy` needs admin
|
||||
from Chrome 130+ ([rookie#50](https://github.com/thewh1teagle/rookie/issues/50)).
|
||||
**Firefox has no ABE** (Mozilla treats local file-access attackers as out-of-scope),
|
||||
so its cookies stay readable by any user-space process — hence the Windows Firefox
|
||||
recommendation. The library uses `rookiepy` (~16 browsers) and reshapes the result
|
||||
via `_auth/cookies.py::convert_rookiepy_cookies_to_storage_state` into a
|
||||
Playwright-compatible `storage_state.json`, indistinguishable downstream from a
|
||||
Playwright-minted one. Extraction asks for the full multi-domain set
|
||||
(`ALLOWED_COOKIE_DOMAINS + GOOGLE_REGIONAL_CCTLDS`) because dropping any one breaks
|
||||
specific paths (e.g. losing `.notebooklm.google.com` cookies breaks artifact
|
||||
downloads).
|
||||
|
||||
### 2.5 Three timers people confuse
|
||||
|
||||
| Timer | Magnitude | Lives in | Meaning |
|
||||
|---|---|---|---|
|
||||
| **`*PSIDTS` rotation cadence** | ~600 s | Google's identity surface | Recommended active-client refresh interval (`["identity.hfcr",600]`). Not a hard rejection TTL — prior values stay valid much longer on stable profiles. |
|
||||
| **`*SIDCC` sliding window** | ~5 min | Google's RPC surface | A different cookie family; rotates on nearly every request; not load-bearing for our auth. |
|
||||
| **Client-side rotation throttle** | 60 s | `_auth/keepalive.py` | Don't fire two `RotateCookies` POSTs within a minute (avoids 429). Unrelated to how often Google *requires* rotation. |
|
||||
|
||||
Reports that "cookies are expiring faster" usually trace to the session entering a
|
||||
risk-flagged state (§3.1) or to the rotation mechanism failing until `*SID` finally
|
||||
ages out — not to a shorter hard rejection TTL.
|
||||
|
||||
### 2.6 Domain tiering: REQUIRED vs OPTIONAL cookie domains
|
||||
|
||||
Not every Google cookie a logged-in browser holds is load-bearing for NotebookLM.
|
||||
The library splits the cookie-source domain list into two tiers
|
||||
(`_auth/cookie_policy.py`):
|
||||
|
||||
| Tier | Constant | Domains | Extracted by default |
|
||||
|---|---|---|:-:|
|
||||
| **REQUIRED** | `REQUIRED_COOKIE_DOMAINS` | `.google.com`, `notebooklm.google.com` (+ regional ccTLDs), `accounts.google.com`, `.googleusercontent.com`, `drive.google.com` | ✅ |
|
||||
| **OPTIONAL** | `OPTIONAL_COOKIE_DOMAINS_BY_LABEL` | `youtube`, `docs`, `myaccount`, `mail` | ❌ (opt-in via `--include-domains=<label>[,…]` or `=all`) |
|
||||
|
||||
The REQUIRED tier is exactly the set traced through every exercised code path (API
|
||||
host, identity carriers, authenticated media downloads, Drive-source ingest);
|
||||
removing any one breaks an observed flow. **Data minimization** motivates the split:
|
||||
`storage_state.json` is a high-value target, and the OPTIONAL tier carries cookies
|
||||
that would let an attacker read Gmail / Drive / YouTube — none of which any
|
||||
NotebookLM path needs. The control is enforced at **extraction time** (what
|
||||
`rookiepy.load(domains=...)` is asked for), so excluded cookies are never written to
|
||||
disk ([#483](https://github.com/teng-lin/notebooklm-py/pull/483)). Opt in only when a
|
||||
sibling flow needs it — e.g.
|
||||
`notebooklm login --browser-cookies firefox --include-domains=youtube,docs`.
|
||||
|
||||
---
|
||||
|
||||
## 3 · Threat model
|
||||
|
||||
### 3.1 What kills a session in practice
|
||||
|
||||
In rough order of likelihood:
|
||||
|
||||
1. **`*PSIDTS` rotation drift.** Cookies on disk go stale because nothing rotates
|
||||
them. Any RPC after the grace period fails with a redirect to
|
||||
`accounts.google.com/v3/signin/…`. **The dominant failure mode for unattended
|
||||
use** — and the one the recovery ladder exists to defeat.
|
||||
2. **Risk-scored revalidation.** Google flags the access pattern (new IP, no
|
||||
fingerprint, suspicious cadence, geography mismatch) and forces full re-auth.
|
||||
Less predictable; days-to-weeks into a long-running deployment.
|
||||
3. **Password change or manual sign-out** anywhere — invalidates all sessions
|
||||
instantly.
|
||||
4. **Workspace policy timeouts.** Some org admins enforce re-auth intervals; varies
|
||||
by tenant.
|
||||
5. **DBSC enforcement (emerging).** See §2.3. Does not affect non-Chrome HTTP
|
||||
clients today; the long-term threat.
|
||||
|
||||
Cookie decay clocks by class:
|
||||
|
||||
| Cookie | Rotation / expiry signal | Lifecycle |
|
||||
|---|---|---|
|
||||
| `__Secure-1PSIDTS` / `*-3PSIDTS` | Recommended cadence ~600 s (`["identity.hfcr",600]`); not a hard TTL | Refreshed opportunistically; stale values work for hours-to-days, then drift into sign-in redirects |
|
||||
| `SIDCC` / `__Secure-*SIDCC` | ~5 min sliding window | Ephemeral; generally not load-bearing for auth |
|
||||
| `SID`, `HSID`, `SSID`, `APISID`, `SAPISID` (+ `__Secure-` cousins) | Months → ~1 year | Long-lived identity; not rotated by us |
|
||||
| `OSID`, `__Secure-OSID` | Per-product session | Re-issued on each sign-in |
|
||||
| `LSID`, `__Host-*LSID`, `__Host-GAPS` | Long-lived | Identity-service / anti-takeover cookies |
|
||||
|
||||
### 3.2 Internal persistence hazards (pointer)
|
||||
|
||||
A separate failure class is easy to misattribute to Google: the library corrupting
|
||||
its own cookie state during the read-merge-write cycle. Historically several such
|
||||
hazards existed (a stale-in-memory-clobbers-fresh-disk race, `(name, domain)`
|
||||
path-collapse, sibling-domain allow-list asymmetry, round-trip attribute erosion).
|
||||
**All of them are resolved in-tree** — the persistence path is now snapshot/delta,
|
||||
CAS-guarded, cross-process flocked, and fully `(name, domain, path)`-aware. If users
|
||||
report cookies "expiring fast", walk the
|
||||
[diagnostic checklist](#a2--diagnosing-cookies-expire-fast) in the Appendix (which
|
||||
also records the historical hazards and their fixes) before assuming Google changed
|
||||
anything.
|
||||
|
||||
### 3.3 Empirical cookie requirements
|
||||
|
||||
Which cookies does Google *actually* require? This backs the library's two-tier
|
||||
`_validate_required_cookies()` pre-flight (see `_auth/cookies.py` —
|
||||
`MINIMUM_REQUIRED_COOKIES` and `_has_valid_secondary_binding()` for the
|
||||
authoritative values; the historical permissive `{"SID"}` check was replaced in
|
||||
[#371](https://github.com/teng-lin/notebooklm-py/issues/371)).
|
||||
|
||||
Method: take a known-good `storage_state.json`, drop one or two cookies at a time,
|
||||
run `notebooklm list`, and record whether Google accepts the call or redirects to
|
||||
login. Single-cookie removal is highly recoverable — every cookie except `SID` can
|
||||
be dropped individually with the call still succeeding (Google reissues most of
|
||||
them mid-call). Pair-wise removal exposes a precise accept-rule.
|
||||
|
||||
**The accept-rule model.** Google accepts the NotebookLM homepage GET when both
|
||||
hold:
|
||||
|
||||
1. **Identity present:** `SID` is valid, and `__Secure-1PSIDTS` is either directly
|
||||
present or recoverable via a `RotateCookies` POST — which itself requires the
|
||||
full ambient cookie set to authenticate.
|
||||
2. **At least one secondary binding present:** `OSID`, OR both `APISID` and
|
||||
`SAPISID`.
|
||||
|
||||
| Variant | `SID` | `OSID` | `APISID+SAPISID` | `__Secure-1PSIDTS` (or recoverable) | Result |
|
||||
|---|:-:|:-:|:-:|:-:|:-:|
|
||||
| Baseline | ✓ | ✓ | ✓ | ✓ | OK |
|
||||
| Drop `__Secure-1PSIDTS` only | ✓ | ✓ | ✓ | recoverable | OK |
|
||||
| Drop `__Secure-1PSIDTS` + any one other | ✓ | ✓ | ✓ | broken (mint POST fails) | FAIL |
|
||||
| Drop `OSID` only | ✓ | ✗ | ✓ | ✓ | OK (AP\*SID path) |
|
||||
| Drop `APISID + SAPISID` | ✓ | ✓ | ✗ | ✓ | OK (OSID path) |
|
||||
| Drop `APISID + OSID` | ✓ | ✗ | ✗ | ✓ | FAIL |
|
||||
| Drop `SAPISID + OSID` | ✓ | ✗ | ✗ | ✓ | FAIL |
|
||||
|
||||
Before #371 the library trusted any storage with `SID` present, which let
|
||||
Google-rejected cookie sets reach the wire — the "auth expires immediately after
|
||||
`notebooklm login`" pattern
|
||||
([#133](https://github.com/teng-lin/notebooklm-py/issues/133),
|
||||
[#332](https://github.com/teng-lin/notebooklm-py/issues/332)). The pre-flight now
|
||||
catches it with a two-tier check:
|
||||
|
||||
```python
|
||||
MINIMUM_REQUIRED_COOKIES = {"SID", "__Secure-1PSIDTS"} # Tier 1: raise
|
||||
|
||||
def _has_valid_secondary_binding(cookie_names: set[str]) -> bool: # Tier 2: warn
|
||||
if "OSID" in cookie_names:
|
||||
return True
|
||||
return {"APISID", "SAPISID"} <= cookie_names
|
||||
```
|
||||
|
||||
Tier 1 raises on unambiguous evidence; Tier 2 warns once per process so partial
|
||||
extractions surface without breaking edge-case flows (e.g. Workspace SSO) that
|
||||
haven't been ablated.
|
||||
|
||||
**Caveats.** These observations came from a single non-Workspace, stable-IP
|
||||
profile, testing `notebooks.list`. Workspace accounts may have different
|
||||
accept-rules. This is a model fit, not a confirmed server mechanism, and the
|
||||
freshness clock (§3.1) still applies on top of it — a session with a valid
|
||||
accept-tuple can still be killed by Google's risk model.
|
||||
|
||||
---
|
||||
|
||||
## 4 · The recovery ladder
|
||||
|
||||
The library escalates progressively as cheaper mechanisms fail (see the ladder
|
||||
table in the [TL;DR](#tldr)). Each layer is a fallback for the one below it: L1/L2
|
||||
are HTTP-only and cheap; L3 drives a browser and only helps if the profile's Google
|
||||
session is still alive; **L4 re-mints from a durable master token with no browser —
|
||||
the best unattended recovery**; L5 delegates policy to the operator; L6 is manual;
|
||||
L7 is proactive scheduling for idle profiles.
|
||||
|
||||
### 4.1 L1 — per-call `RotateCookies` POST
|
||||
|
||||
Fires inside the token-fetch path on every CLI invocation and client open. A best-
|
||||
effort `POST https://accounts.google.com/RotateCookies` that mints a fresh
|
||||
`*PSIDTS`. Default ON; disable with `NOTEBOOKLM_DISABLE_KEEPALIVE_POKE=1`. Wrapped
|
||||
in three concentric guards (disk-mtime fast-path → in-process `asyncio.Lock` +
|
||||
per-profile monotonic timestamp → cross-process non-blocking flock) so an L1 caller,
|
||||
an L2 loop, and a fan-out of parallel CLI invocations keyed to the same
|
||||
`storage_state.json` don't stampede the endpoint into a 429. Mechanics in
|
||||
[§5](#5--the-rotatecookies-primitive).
|
||||
|
||||
### 4.2 L2 — background keepalive task
|
||||
|
||||
`NotebookLMClient(keepalive=N)` starts an `asyncio.Task` that pokes
|
||||
`RotateCookies` every N seconds (floor 60 s) while the client is open. Self-paced,
|
||||
so it bypasses the L1 fast-path guards but still performs the atomic per-profile
|
||||
claim, so a sibling L1 poke sees the in-flight rotation and skips. Covers agents,
|
||||
MCP servers, and long-running workers.
|
||||
|
||||
### 4.3 L3 — headless re-auth / CDP attach
|
||||
|
||||
When the homepage GET 302s to the Google login page, the first-party cookies are
|
||||
fully dead and neither L1 nor L2 can help. `refresh_auth(allow_headless=True)` (or
|
||||
`NOTEBOOKLM_HEADLESS_REAUTH=1` for automatic mid-RPC opt-in) drives an unattended
|
||||
headless browser against the **persisted profile that is a sibling of this client's
|
||||
`storage_state.json`** (`<storage_path>/../browser_profile`) — never the ambient
|
||||
profile — to silently re-mint cookies, then reloads them and retries the homepage
|
||||
GET once. Set `NOTEBOOKLM_HEADLESS_REAUTH_CDP_URL=http://127.0.0.1:9222` to attach
|
||||
to an already-running local Chrome instead; non-loopback hosts are refused because
|
||||
a CDP endpoint is account-equivalent. If the profile is missing, Playwright is
|
||||
unavailable, env-var auth has no writeable file, or the browser session is also
|
||||
dead, the original auth-expiry error stands. Owner:
|
||||
`_auth/headless_reauth.py`; integration point:
|
||||
`_auth/session.py::refresh_auth_session`.
|
||||
|
||||
### 4.4 L4 — master-token re-mint
|
||||
|
||||
When the profile holds a `master_token.json` (written by
|
||||
`notebooklm login --master-token`, the `[headless]` extra), a fully-expired session
|
||||
re-mints **in process, with no browser** — the recovery the `RotateCookies` /
|
||||
headless-browser ladder can't provide off-device.
|
||||
|
||||
- **Credential.** A durable Google **master token** (`aas_et/…`), obtained once from
|
||||
`accounts.google.com/EmbeddedSetup` and stored `0600`. It mints fresh web cookies
|
||||
on demand (`perform_oauth → OAuthLogin?issueuberauth=1 → MergeSession`) and
|
||||
survives password changes until explicitly revoked. It also bootstraps the initial
|
||||
`storage_state.json`.
|
||||
- **Where it fires.** `_auth/session.py::_try_master_token_reauth`, as **layer 4 of
|
||||
`refresh_auth_session`** — only after L1 (homepage), L2 (`RotateCookies`), and L3
|
||||
(headless browser) are exhausted. It mints a new session, persists it (replacing
|
||||
the dead cookies under the storage lock), reloads the jar into the live HTTP
|
||||
client, and retries the homepage GET once. Reached through the
|
||||
`AuthRefreshCoordinator` single-flight, so concurrent RPCs coalesce **one**
|
||||
re-mint.
|
||||
- **Cold start.** A session already dead at process start is recovered by
|
||||
`notebooklm login --master-token-refresh` (or the next bootstrap); the in-process
|
||||
layer-4 covers the mid-session case long-lived workers hit.
|
||||
- **PSIDTS interaction.** A re-mint yields `SID`+`APISID`+`SAPISID` but not
|
||||
`__Secure-1PSIDTS`; the mint itself fires one best-effort `RotateCookies` POST to
|
||||
add it, and the inline PSIDTS recovery
|
||||
([§5.4](#54-inline-__secure-1psidts-cold-start-recovery)) mints it from the
|
||||
secondary binding on reload if Google withheld it — so L1 keepalive then proceeds
|
||||
normally on the fresh session.
|
||||
- **Security & limits.** The master token is full-account and infostealer-grade —
|
||||
dedicated/throwaway account only, never logged or committed. Each re-mint is a new
|
||||
session, so one account is **single-consumer**: concurrent re-mints can invalidate
|
||||
each other's `SID`. DBSC is the standing risk; re-mint is itself the mitigation
|
||||
while it isn't enforced. See [ADR-0023](adr/0023-master-token-headless-auth.md).
|
||||
|
||||
### 4.5 L5–L7
|
||||
|
||||
- **L5 — `NOTEBOOKLM_REFRESH_CMD`.** An operator-supplied recovery command run on
|
||||
an auth-expiry signal, with one retry. Same-loop callers coalesce on one
|
||||
subprocess and cancellation-safe. See [§6.2](#62-notebooklm_refresh_cmdcommand-line-l5).
|
||||
- **L6 — `notebooklm login`.** Baseline manual recovery: interactive browser
|
||||
sign-in, or `--browser-cookies <browser>` extraction (§2.4).
|
||||
- **L7 — `notebooklm auth refresh`.** A one-shot token fetch driven by cron /
|
||||
launchd / systemd / Task Scheduler / k8s CronJob, for profiles idle between Python
|
||||
runs. Recommended cadence 15–20 min.
|
||||
|
||||
---
|
||||
|
||||
## 5 · The `RotateCookies` primitive
|
||||
|
||||
The L1/L2 rotation POST and the L4 re-mint's PSIDTS top-up all share one endpoint.
|
||||
|
||||
### 5.1 The endpoint
|
||||
|
||||
```
|
||||
POST https://accounts.google.com/RotateCookies
|
||||
Content-Type: application/json
|
||||
Origin: https://accounts.google.com
|
||||
|
||||
[000,"-0000000000000000000"]
|
||||
```
|
||||
|
||||
The body is a JSPB (array-shaped) sentinel. `000` is `0` written with leading zeros
|
||||
(valid in Google's JSPB parser, invalid in strict JSON); `"-0000000000000000000"`
|
||||
is a sentinel meaning "I have no prior `__Secure-1PSIDTS`, mint a fresh one from the
|
||||
persistent identity (`SID`/`PSID`) alone." The pattern is borrowed from
|
||||
[`HanaokaYuzu/Gemini-API`](https://github.com/HanaokaYuzu/Gemini-API/blob/master/src/gemini_webapi/utils/rotate_1psidts.py),
|
||||
which has run it in production at scale.
|
||||
|
||||
### 5.2 The successful response
|
||||
|
||||
```
|
||||
HTTP/1.1 200 OK
|
||||
Set-Cookie: __Secure-1PSIDTS=<new>; Domain=.google.com; Secure; HttpOnly
|
||||
Set-Cookie: __Secure-3PSIDTS=<new>; Domain=.google.com; Secure; HttpOnly
|
||||
Set-Cookie: SIDCC=<new>; Domain=.google.com; Secure
|
||||
…
|
||||
|
||||
)]}' [["identity.hfcr",600],["di",<integer>]]
|
||||
```
|
||||
|
||||
`)]}'` is Google's anti-XSSI prefix. `["identity.hfcr",600]` declares the
|
||||
recommended next-rotation interval (600 s); `["di",N]` is an opaque session counter.
|
||||
`_auth/storage.py::save_cookies_to_storage` captures the rotated `Set-Cookie`
|
||||
headers and persists them atomically.
|
||||
|
||||
Why `RotateCookies` and not the older `CheckCookie` GET: `RotateCookies` rotates
|
||||
`*PSIDTS` unconditionally for **both** browser-bound (Playwright) and unbound
|
||||
(Firefox-extracted) sessions, whereas `CheckCookie` only rotated for unbound ones.
|
||||
Historical detail in the [Appendix](#a1--rotatecookies-vs-checkcookie).
|
||||
|
||||
### 5.3 Rate limiting and the three-guard throttle
|
||||
|
||||
Hammering `RotateCookies` triggers HTTP 429. The mitigation is a 60-second floor,
|
||||
enforced by three concentric guards (`_auth/keepalive.py`,
|
||||
[#346](https://github.com/teng-lin/notebooklm-py/pull/346) +
|
||||
[#348](https://github.com/teng-lin/notebooklm-py/pull/348)):
|
||||
|
||||
1. **Disk-mtime fast-path** — skip without any lock if `storage_state.json` was
|
||||
rewritten within `_KEEPALIVE_RATE_LIMIT_SECONDS` (60 s). A 2 s tolerance absorbs
|
||||
filesystem mtime granularity; a far-future mtime is treated as not-recent.
|
||||
2. **In-process throttle** — inside an `asyncio.Lock` keyed by
|
||||
`(running loop, storage_path)`, re-check mtime plus a per-profile monotonic
|
||||
timestamp stamped under a `threading.Lock`. Deduplicates an `asyncio.gather`
|
||||
fan-out; the timestamp is bumped *before* the network await so a hung
|
||||
`accounts.google.com` doesn't make N callers each wait the full timeout.
|
||||
3. **Cross-process non-blocking flock** (`.storage_state.json.rotate.lock` via
|
||||
`LOCK_NB`) — if another process holds it, skip; they're rotating now. Distinct
|
||||
from the save-path lock so a long save never blocks rotations. Locks **fail
|
||||
open** on read-only / NFS filesystems: rotation proceeds rather than wedging.
|
||||
|
||||
Together the three guards cover sequential CLI invocations (mtime fast-path), an
|
||||
`asyncio.gather` fan-out from one process (in-process lock + stamp), an L1 caller
|
||||
racing the L2 loop (per-profile monotonic stamp), and simultaneous processes
|
||||
(cross-process flock). The per-`(loop, profile)` lock dict is a `WeakKeyDictionary`
|
||||
keyed on the loop object, so a short-lived `asyncio.run()` loop's inner dict is
|
||||
reclaimed on GC.
|
||||
|
||||
### 5.4 Inline `__Secure-1PSIDTS` cold-start recovery
|
||||
|
||||
When a profile has the persistent `__Secure-1PSID` but no transient
|
||||
`__Secure-1PSIDTS` (a common cold-start snapshot), `_recover_psidts_inline`
|
||||
(`_auth/psidts_recovery.py`) makes a preflight `RotateCookies` POST during client
|
||||
startup to mint the missing cookie before the first RPC. It fires only when
|
||||
`__Secure-1PSID` is present and `__Secure-1PSIDTS` is missing, honors
|
||||
`NOTEBOOKLM_DISABLE_KEEPALIVE_POKE=1`, and uses a cross-process flock
|
||||
(`psidts_recovery.lock`) so concurrent cold-start processes don't fan out identical
|
||||
recovery calls. This is what lets the L4 re-mint (which produces `SID` +
|
||||
`APISID`/`SAPISID` but not `*PSIDTS`) heal into a complete jar on reload. See
|
||||
[ADR-0013](adr/0013-composable-session-capabilities.md#consequences).
|
||||
|
||||
---
|
||||
|
||||
## 6 · Operational levers (environment variables)
|
||||
|
||||
Auth-refresh env vars live under `src/notebooklm/_auth/` and are re-exported through
|
||||
the public `notebooklm.auth` facade where compatibility requires it. See also
|
||||
[configuration.md#environment-variables](configuration.md#environment-variables).
|
||||
|
||||
### 6.1 `NOTEBOOKLM_DISABLE_KEEPALIVE_POKE=1`
|
||||
|
||||
Disables the `RotateCookies` POST entirely. Both L1 and L2 honor it (the L2 task
|
||||
still wakes on its interval — only the network call becomes a no-op; pass
|
||||
`keepalive=None` to stop the loop itself). Set it on restricted networks that block
|
||||
outbound POSTs to `accounts.google.com`, for regression triage, or in test
|
||||
environments that mock the auth surface.
|
||||
|
||||
### 6.2 `NOTEBOOKLM_REFRESH_CMD=<command-line>` (L5)
|
||||
|
||||
Reactive recovery hook (merged in
|
||||
[#336](https://github.com/teng-lin/notebooklm-py/pull/336), hardened to
|
||||
`shell=False` by default in
|
||||
[#475](https://github.com/teng-lin/notebooklm-py/pull/475); owner
|
||||
`_auth/refresh.py`). When token fetch fails with an auth-expiry signal (the
|
||||
"`Authentication expired or invalid`" / `accounts.google.com` redirect), the
|
||||
library:
|
||||
|
||||
1. Parses the command with `shlex.split` (POSIX) / `CommandLineToArgvW` (Windows)
|
||||
and runs it via `subprocess.run(argv, shell=False, …)` with a 60 s timeout. Set
|
||||
`NOTEBOOKLM_REFRESH_CMD_USE_SHELL=1` to opt back into `shell=True` (a `WARNING`
|
||||
is logged each invocation).
|
||||
2. Sets `NOTEBOOKLM_REFRESH_PROFILE` / `NOTEBOOKLM_REFRESH_STORAGE_PATH` in the
|
||||
child env so the script knows which profile to refresh.
|
||||
3. Sets `_NOTEBOOKLM_REFRESH_ATTEMPTED=1` to prevent recursive refresh loops.
|
||||
4. Scrubs `NOTEBOOKLM_AUTH_JSON` from the child env (credential-equivalent; the
|
||||
script gets the on-disk path via step 2 instead).
|
||||
5. Reloads cookies from `storage_state.json` and replays the token fetch once.
|
||||
|
||||
> **SECURITY — inherited environment.** The refresh command inherits the **full
|
||||
> parent environment** (minus the `NOTEBOOKLM_AUTH_JSON` scrub) so it can find
|
||||
> `PATH`/`HOME`/proxy settings and re-invoke this library. There is deliberately no
|
||||
> allowlist. Any other secret in the launching shell is inherited by the command and
|
||||
> every grandchild, and is visible via `/proc/<pid>/environ` to the same UID.
|
||||
> Operators MUST NOT keep unrelated secrets in the launching environment
|
||||
> ([#1274](https://github.com/teng-lin/notebooklm-py/issues/1274)).
|
||||
|
||||
Same-loop fan-out coalesces on one shielded in-flight subprocess, so cancellation of
|
||||
one caller doesn't cancel the shared command; cross-loop coalescing is best-effort
|
||||
(cross-loop client reuse is unsupported per
|
||||
[ADR-0004](adr/0004-loop-affinity-contract.md)).
|
||||
|
||||
This is **orthogonal to L1–L4**: those proactively keep the session fresh (L1/L2) or
|
||||
re-mint it in-process (L3/L4), while `NOTEBOOKLM_REFRESH_CMD` runs only after auth
|
||||
has already fully expired — useful for password-change / manual-sign-out recovery or
|
||||
a custom CookieCloud / browser-cookie re-extract flow. Common shapes:
|
||||
|
||||
```bash
|
||||
export NOTEBOOKLM_REFRESH_CMD='notebooklm login --browser-cookies firefox'
|
||||
export NOTEBOOKLM_REFRESH_CMD='/opt/scripts/pull-cookies-from-cloud.sh'
|
||||
```
|
||||
|
||||
The library does not validate the command's output; the operator must ensure it
|
||||
produces a valid `storage_state.json`.
|
||||
|
||||
### 6.3 `NOTEBOOKLM_HEADLESS_REAUTH=1` and `NOTEBOOKLM_HEADLESS_REAUTH_CDP_URL` (L3)
|
||||
|
||||
Opt into automatic L3 headless re-auth during mid-RPC refresh (explicit
|
||||
`await client.refresh_auth(allow_headless=True)` needs no env var). The CDP URL, if
|
||||
set, attaches to an already-running loopback Chrome instead of launching the stored
|
||||
profile; non-loopback hosts are refused. Details in [§4.3](#43-l3--headless-re-auth--cdp-attach).
|
||||
|
||||
---
|
||||
|
||||
## 7 · Canaries and signals
|
||||
|
||||
Tripwires that would signal the threat model shifting:
|
||||
|
||||
| Signal | What it means | Action |
|
||||
|---|---|---|
|
||||
| `RotateCookies` returns 401 in production | DBSC extended to non-Chrome paths for some accounts | Harden the L3 CDP arm; steer users to `NOTEBOOKLM_HEADLESS_REAUTH_CDP_URL` or an L5 CookieCloud flow |
|
||||
| `RotateCookies` returns 200 but no `*PSIDTS` in `Set-Cookie` | Silent failure — cookies on disk aren't rotating | WARN + alert; manual re-auth required |
|
||||
| Gemini-API's bare-sentinel rotation reported decaying under DBSC | Upstream canary for the shared primitive | Assess whether our user base is affected; plan a mitigation |
|
||||
| Chrome macOS DBSC GA announced | macOS users start getting DBSC enrollment | Several months' warning before consumer accounts may be enforced |
|
||||
| Workspace session-binding leaves beta | More org admins will enable it | Document explicit non-support more clearly |
|
||||
|
||||
---
|
||||
|
||||
## 8 · References
|
||||
|
||||
**Project peers**
|
||||
|
||||
- [HanaokaYuzu/Gemini-API](https://github.com/HanaokaYuzu/Gemini-API) — reference
|
||||
for `RotateCookies` rotation
|
||||
([source](https://github.com/HanaokaYuzu/Gemini-API/blob/master/src/gemini_webapi/utils/rotate_1psidts.py)).
|
||||
Our L1 mirrors it; its lack of any reactive fallback is the gap our L3/L4/L5 close.
|
||||
- [easychen/CookieCloud](https://github.com/easychen/CookieCloud) +
|
||||
[PyCookieCloud](https://github.com/lupohan44/PyCookieCloud) — DBSC-immune cookie
|
||||
federation, a viable `NOTEBOOKLM_REFRESH_CMD` source; no in-tree client.
|
||||
- [dsdanielpark/Bard-API](https://github.com/dsdanielpark/Bard-API) (archived) — the
|
||||
cautionary tale: reactive/manual-only cookie management proved untenable.
|
||||
|
||||
**Cookie extraction**
|
||||
|
||||
- [`thewh1teagle/rookie`](https://github.com/thewh1teagle/rookie) (rookiepy),
|
||||
[`borisbabic/browser_cookie3`](https://github.com/borisbabic/browser_cookie3),
|
||||
[`n8henrie/pycookiecheat`](https://github.com/n8henrie/pycookiecheat)
|
||||
|
||||
**DBSC**
|
||||
|
||||
- [Google DBSC announcement](https://blog.google/security/protecting-cookies-with-device-bound-session-credentials/),
|
||||
[Chrome DBSC Windows GA](https://developer.chrome.com/blog/dbsc-windows-announcement),
|
||||
[W3C DBSC spec](https://w3c.github.io/webappsec-dbsc/),
|
||||
[Workspace session-binding](https://knowledge.workspace.google.com/admin/security/prevent-cookie-theft-with-session-binding)
|
||||
|
||||
**In-repo**
|
||||
|
||||
- [ADR-0023 — master-token headless auth](adr/0023-master-token-headless-auth.md)
|
||||
- [ADR-0013 — composable session capabilities](adr/0013-composable-session-capabilities.md)
|
||||
- [ADR-0004 — loop-affinity contract](adr/0004-loop-affinity-contract.md)
|
||||
- [troubleshooting.md#authentication-errors](troubleshooting.md#authentication-errors),
|
||||
[installation.md#d-headless-server-or-ci](installation.md#d-headless-server-or-ci),
|
||||
[configuration.md#environment-variables](configuration.md#environment-variables)
|
||||
|
||||
---
|
||||
|
||||
## Appendix: field notes & historical findings
|
||||
|
||||
Condensed war-stories and ruled-out experiments, kept for triage context. None of
|
||||
this is required to operate the library.
|
||||
|
||||
### A1 · `RotateCookies` vs `CheckCookie`
|
||||
|
||||
The original L1 mechanism used
|
||||
`GET accounts.google.com/CheckCookie?continue=…notebooklm.google.com/`, relying on
|
||||
a redirect chain that *might* pass through `accounts.youtube.com/SetSID` and set a
|
||||
fresh `*PSIDTS`. Field probing showed this rotates `*PSIDTS` only for
|
||||
Firefox-extracted (unbound) profiles — a 3-hop chain including `SetSID` — and **not**
|
||||
for Playwright-extracted (bound) profiles, whose 2-hop chain has no `SetSID` step
|
||||
and no `*PSIDTS` in any `Set-Cookie`. The bound-session poke still touched the
|
||||
identity surface and observably extended server-side session validity, but did not
|
||||
rotate the cookie. A direct `RotateCookies` POST removes the discretion: it rotates
|
||||
unconditionally for both session types, at a ~100% success rate in all field
|
||||
captures, with no DBSC challenge on the unsigned endpoint.
|
||||
|
||||
### A2 · Diagnosing "cookies expire fast"
|
||||
|
||||
The persistence pipeline could historically corrupt its own cookie state. Before
|
||||
assuming Google changed anything:
|
||||
|
||||
1. **Compare `__Secure-1PSIDTS` on disk before/after a `notebooklm` call** spaced
|
||||
> 60 s apart with no other writer. No change ⇒ rotation isn't firing — check
|
||||
`NOTEBOOKLM_DISABLE_KEEPALIVE_POKE` and the mtime guard.
|
||||
2. **With multiple processes sharing the file**, run at `NOTEBOOKLM_LOG_LEVEL=DEBUG`
|
||||
and look for "Keepalive RotateCookies skipped: storage refreshed before flock
|
||||
acquired" — that means the guards are working.
|
||||
3. **Check `storage_state.json` mtime cadence** — hours-old mtime after active
|
||||
sessions means rotation isn't sticking.
|
||||
4. **Only after the above**, investigate Google-side causes (risk-scoring, Workspace
|
||||
policy, DBSC).
|
||||
|
||||
**Historical persistence hazards (all resolved in-tree)** — recorded so older bug
|
||||
reports still make sense:
|
||||
|
||||
- **Stale-in-memory clobbers fresh-disk ("few-hours" pattern).** A `keepalive=None`
|
||||
process could write its stale in-memory `*PSIDTS` over a fresher value a sibling
|
||||
rotated. Resolved by open-time snapshot + write-only-deltas with value-CAS guards
|
||||
(`_cookie_persistence.py`), on top of the cross-process flock
|
||||
([#344](https://github.com/teng-lin/notebooklm-py/pull/344) +
|
||||
[#361](https://github.com/teng-lin/notebooklm-py/pull/361)). Still, prefer
|
||||
`keepalive=N` or a single cron-driven rotator.
|
||||
- **`(name, domain)` path-collapse.** The persistence merge is now fully
|
||||
`(name, domain, path)`-aware end-to-end, so path-scoped variants survive a
|
||||
load→save round-trip ([#369](https://github.com/teng-lin/notebooklm-py/pull/369)).
|
||||
- **Sibling-domain allow-list asymmetry.** The auth-jar and persistence filters were
|
||||
collapsed into one canonical `_is_allowed_cookie_domain` with siblings covered
|
||||
symmetrically ([#360](https://github.com/teng-lin/notebooklm-py/issues/360)).
|
||||
- **Round-trip attribute erosion.** Both load paths now build a faithful
|
||||
`http.cookiejar.Cookie` preserving `path` / `secure` / `httpOnly` across cycles
|
||||
([#365](https://github.com/teng-lin/notebooklm-py/pull/365)), keeping `__Host-`
|
||||
invariants intact.
|
||||
- **`expires=-1` flattens age.** `*PSIDTS` rotations arrive without `Max-Age` and are
|
||||
stored as session cookies, so on-disk age is unknowable — the only staleness signal
|
||||
is the file mtime. By design, not a bug.
|
||||
|
||||
Across the OSS ecosystem this is the most defensive cookie-persistence implementation
|
||||
surveyed: peers (Gemini-API, yt-dlp, CookieCloud) are "last writer wins" with no
|
||||
flock, no atomic replace, and full-jar overwrite. Our threat model (long-lived
|
||||
clients + cron `auth refresh` + parallel CLI invocations on one file) genuinely needs
|
||||
the snapshot/delta/CAS/flock defenses.
|
||||
|
||||
### A3 · Ruled-out experiments
|
||||
|
||||
Investigated and rejected; documented so contributors don't re-investigate:
|
||||
|
||||
- **`undetected-chromedriver` / `selenium-stealth` for Google login** — WebDriver
|
||||
stealth loses to Google's signal-fusion model; login has been repeatedly broken
|
||||
across Chrome bumps. Don't use for Google flows.
|
||||
- **`puppeteer-extra-plugin-stealth` / `playwright-stealth`** — patches fingerprint
|
||||
leaks only, not TLS / IP reputation / behavioral signals; works for resumed
|
||||
sessions, fails for fresh `accounts.google.com` sign-in.
|
||||
- **Persistent Playwright headless context as a keepalive daemon** — fragile against
|
||||
known Playwright bugs (cookies missing in `launch_persistent_context`, profile-DB
|
||||
corruption in long-lived contexts). Prefer CDP-attach (the L3 arm) if a
|
||||
headless-browser path is needed.
|
||||
- **Client-side DBSC implementation** — impossible from Python: the W3C spec is
|
||||
built around a TPM-bound key and platform attestation Chrome implements, with no
|
||||
extension point for a non-browser client. If DBSC extends to non-Chrome paths, the
|
||||
escape is to parasitize a DBSC-enrolled Chrome via the L3 CDP arm, or source
|
||||
cookies via an operator-provided `NOTEBOOKLM_REFRESH_CMD`.
|
||||
- **Reading the Chrome cookie DB on Chrome 127+** — App-Bound Encryption makes this
|
||||
admin-or-bust on Windows; the yt-dlp ecosystem has converged on Firefox as the
|
||||
only reliable `--cookies-from-browser` source. Infostealer-adjacent ABE-bypass
|
||||
forks are inappropriate to ship. Document `--browser-cookies firefox` as the
|
||||
Windows path.
|
||||
|
||||
### A4 · Anti-pattern — persisting `storage_state` on a redirect-to-login
|
||||
|
||||
If you write your own Playwright keepalive instead of using `notebooklm auth
|
||||
refresh` or `keepalive=N`, the most damaging mistake is calling
|
||||
`context.storage_state(path=…)` unconditionally at the end of each cycle. When the
|
||||
session has aged out, `page.goto("https://notebooklm.google.com/")` 302s to
|
||||
`accounts.google.com/…/SignIn`, the login page sets a handful of anonymous cookies
|
||||
(`NID`, `OTZ`, `__Host-GAPS`, `_ga`, …), and an unconditional `storage_state`
|
||||
serializes **only those**, dropping every real auth cookie. The next cold start
|
||||
finds a useless file and recovery requires a fresh interactive login. The rule for
|
||||
any wrapper that owns its own persistence: **gate the write on a confirmed-authed
|
||||
page URL, or better, on a successful library API call.**
|
||||
|
||||
```python
|
||||
from notebooklm import NotebookLMClient, AuthError
|
||||
|
||||
async def verify_and_save(context, STORAGE):
|
||||
try:
|
||||
async with NotebookLMClient.from_storage() as client:
|
||||
await client.notebooks.list() # confirms auth
|
||||
except (AuthError, ValueError):
|
||||
return # don't overwrite a good file with a bad jar
|
||||
await context.storage_state(path=STORAGE)
|
||||
```
|
||||
|
||||
The supported keepalive surfaces (`notebooklm auth refresh`, `keepalive=N`) already
|
||||
gate their writes correctly.
|
||||
|
||||
### A5 · Open questions
|
||||
|
||||
- **Exact `*PSIDTS` stale-value acceptance distribution.** Acceptance varies by
|
||||
account, IP, Workspace policy, and extraction quality; longitudinal data would let
|
||||
us tune L2's 60 s floor more precisely.
|
||||
- **DBSC enrollment status for Playwright-launched Chromium.** Assumed non-bound on
|
||||
macOS/Linux (no TPM), possibly bound on Windows; untested.
|
||||
- **Whether `RotateBoundCookies` returns interpretable errors for unsigned
|
||||
attempts** — could let us detect a DBSC enforcement transition proactively rather
|
||||
than reactively.
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
- **2026-07-04 (v0.8.0 rewrite)** — Restructured and trimmed for the v0.8.0
|
||||
release. Now leads with the master-token headless path (L4) as the recommended
|
||||
approach for unattended / headless / server / CI use, adds an early
|
||||
[Available auth methods](#available-auth-methods) comparison and a
|
||||
[Recommended setup](#recommended-setup) section, and consolidates the recovery
|
||||
ladder onto the single canonical **L1–L7** taxonomy shared with
|
||||
troubleshooting.md — reflecting the integrated L3 headless re-auth and L4
|
||||
master-token re-mint. Corrected the prior status blockquote, relabeled
|
||||
browser-cookie extraction (a variant of L6 / a common L5 command, **not** L4),
|
||||
removed the obsolete "L5-A" label, and updated the `NOTEBOOKLM_REFRESH_CMD`
|
||||
scope note to "orthogonal to L1–L4". Dated empirical claims (specific probe
|
||||
runs, hour-counts, month-stamped DBSC timeline) were replaced with durable,
|
||||
timeless statements. The internal cookie-jar-fidelity hazard log, the
|
||||
`RotateCookies`-vs-`CheckCookie` deep dive, the ruled-out experiments, and the
|
||||
ecosystem comparison were compressed into the
|
||||
[Appendix](#appendix-field-notes--historical-findings).
|
||||
- **Earlier revisions (2026-05 → 2026-06)** — initial writeup and field experiment;
|
||||
merged-code sync (L1 `RotateCookies` POST, three-guard throttle); §2 background
|
||||
(cookie taxonomy, rotation model, DBSC, extraction); internal-threat
|
||||
cookie-jar-fidelity analysis; domain tiering (REQUIRED vs OPTIONAL); path-aware
|
||||
persistence follow-ups. Superseded by the v0.8.0 rewrite above; see git history
|
||||
for the full detail.
|
||||
@@ -0,0 +1,433 @@
|
||||
# CLI Exit-Code Convention
|
||||
|
||||
**Status:** Active
|
||||
**Last Updated:** 2026-07-04
|
||||
|
||||
This document defines the exit-code policy for the `notebooklm` CLI. Shell
|
||||
scripts, CI pipelines, and AI-agent automations should rely on these codes for
|
||||
control flow rather than scraping stdout/stderr text — the text is intended for
|
||||
humans and may evolve, but the exit-code contract is stable.
|
||||
|
||||
The companion architectural decision for the `--json` error contract is
|
||||
[ADR-0015](adr/0015-json-envelope-contract-for-post-parse-click-exceptions.md);
|
||||
this document is the surface-level reference for callers, and ADR-0015 is the
|
||||
rationale for the post-parse `ClickException` rules called out below.
|
||||
|
||||
For the canonical implementation, see the `handle_errors` context manager in
|
||||
[`src/notebooklm/cli/error_handler.py`](../src/notebooklm/cli/error_handler.py)
|
||||
— the policy table lives in its docstring and the `KeyboardInterrupt` clause
|
||||
sits immediately below (at the time of writing, around lines 64-67 and :81;
|
||||
rely on the symbol names rather than the line numbers if they drift).
|
||||
|
||||
## Standard exit codes
|
||||
|
||||
| Code | Meaning | When you'll see it |
|
||||
|------|---------|-------------------|
|
||||
| `0` | Success | The command completed and produced its intended effect. |
|
||||
| `1` | User / application error | Validation, authentication, rate limiting, network failure, configuration error, or any `NotebookLMError` raised by the library. |
|
||||
| `2` | System / unexpected error | Unhandled exception (likely a bug). The CLI suggests reporting at the issue tracker. Also used for the `source wait` timeout (see exceptions below). |
|
||||
| `130`| Cancelled by user | The process received `SIGINT` (Ctrl-C). `130 = 128 + signal 2`, the conventional shell value for SIGINT-terminated processes. |
|
||||
|
||||
The policy comment in `error_handler.py` is the source of truth:
|
||||
|
||||
```text
|
||||
Exit codes:
|
||||
1: User/application error (validation, auth, rate limit, etc.)
|
||||
2: System/unexpected error (bugs, unhandled exceptions)
|
||||
130: Keyboard interrupt (128 + signal 2)
|
||||
```
|
||||
|
||||
<a id="exception-exit-code-mapping"></a>
|
||||
|
||||
## Exception → exit-code mapping
|
||||
|
||||
The `handle_errors` context manager wrapping every CLI command translates
|
||||
library exceptions into exit codes. The table below summarises the live
|
||||
mapping in `error_handler.py`:
|
||||
|
||||
| Library exception | JSON `code` | Exit |
|
||||
|---|---|---|
|
||||
| `RateLimitError` | `RATE_LIMITED` | `1` |
|
||||
| `AuthError` | `AUTH_ERROR` | `1` |
|
||||
| `ValidationError` | `VALIDATION_ERROR` | `1` |
|
||||
| `ConfigurationError` | `CONFIG_ERROR` | `1` |
|
||||
| `NetworkError` | `NETWORK_ERROR` | `1` |
|
||||
| `NotebookLimitError` | `NOTEBOOK_LIMIT` | `1` |
|
||||
| `NotFoundError` (and domain `*NotFoundError`) | `NOT_FOUND` | `1` |
|
||||
| `ArtifactTimeoutError` (and its subclasses) | `ARTIFACT_TIMEOUT` | `1` |
|
||||
| `NotebookLMError` (other) | `NOTEBOOKLM_ERROR` | `1` |
|
||||
| `KeyboardInterrupt` | `CANCELLED` | `130` |
|
||||
| Anything else (`Exception`) | `UNEXPECTED_ERROR` | `2` |
|
||||
| Parse-time `click.UsageError` / `click.BadParameter` (Click's parser, before command body runs) | `VALIDATION_ERROR` under `--json`; — in text mode | `2` (under `--json`: typed JSON envelope on stdout, **exit code preserved**; text mode: Click's `Usage:/Error:` on stderr) |
|
||||
| Parse-time `click.ClickException` (other subclasses raised by Click's parser) | `VALIDATION_ERROR` under `--json`; — in text mode | `1` (same: JSON envelope under `--json` with the exit code preserved; native text otherwise) |
|
||||
| Post-parse `ClickException` raised from a command body or service module | `VALIDATION_ERROR` (or another standard code, per the raise site) | `1` (typed JSON envelope under `--json`; see ADR-0015) |
|
||||
|
||||
`click.ClickException` raised by **Click's own parser** is the *parse-time*
|
||||
path: argv parsing decides the command body should not run at all (unknown
|
||||
flag, type-validation failure, missing required argument), so `handle_errors(...)`
|
||||
— the per-command context manager — is never on the stack. But the **root
|
||||
group** (`SectionedGroup.main` in
|
||||
[`src/notebooklm/cli/grouped.py`](../src/notebooklm/cli/grouped.py)) sits
|
||||
*above* `handle_errors`: it runs Click's superclass in non-standalone mode and
|
||||
catches the parse-time `ClickException` itself.
|
||||
|
||||
- **Under `--json`**, the root group emits the typed JSON error envelope on
|
||||
stdout (`{ "error": true, "code": "VALIDATION_ERROR", "message": ... }`,
|
||||
empty stderr), so automation that passed `--json` still gets a parseable
|
||||
document for argv-level failures. The **exit code is preserved** from Click —
|
||||
`2` for `UsageError` / `BadParameter`, `1` for the base `ClickException` and
|
||||
other non-usage subclasses. (Note the envelope `code` is `VALIDATION_ERROR`
|
||||
regardless of which Click subclass fired; only the exit code carries the
|
||||
`2`-vs-`1` distinction.)
|
||||
- **In text mode** (no `--json`), behavior is unchanged: Click renders its own
|
||||
`Usage: ... / Error: ...` message on stderr and exits with that same
|
||||
`exc.exit_code` (`2` / `1`).
|
||||
|
||||
This supersedes the original ADR-0015 §1 stance that "no JSON envelope is
|
||||
emitted at parse time"; see the amendment note in
|
||||
[ADR-0015](adr/0015-json-envelope-contract-for-post-parse-click-exceptions.md).
|
||||
|
||||
`ClickException`-subclass failures raised from inside a **command body or
|
||||
its service-layer code** are *post-parse*: argv parsing succeeded, the
|
||||
command function entered, and `--json`'s value (if any) is bound on the
|
||||
Click context. These failures route through `output_error(...)` (the
|
||||
canonical envelope emitter) and exit `1` with the typed JSON error
|
||||
envelope under `--json` or a plain stderr message in text mode. The
|
||||
typical code is `VALIDATION_ERROR`. New command/service code MUST NOT
|
||||
raise `ClickException` for post-parse validation failures unless the call site
|
||||
is an intentional CLI input-validation boundary marked inline with
|
||||
`# cli-input-validation: <reason>`. Raw `SystemExit` sites outside the central
|
||||
handler are similarly marked with `# cli-raw-exit: <reason>`. The guardrail is
|
||||
[`tests/_guardrails/test_error_handler_allowlist.py`](../tests/_guardrails/test_error_handler_allowlist.py);
|
||||
see [ADR-0015](adr/0015-json-envelope-contract-for-post-parse-click-exceptions.md)
|
||||
for the contract and rationale.
|
||||
|
||||
## JSON output mode (`--json`)
|
||||
|
||||
When a command supports `--json` and the flag is set,
|
||||
errors are emitted as a JSON document on stdout *and* the exit code still
|
||||
applies. The shape is:
|
||||
|
||||
```json
|
||||
{
|
||||
"error": true,
|
||||
"code": "RATE_LIMITED",
|
||||
"message": "Error: Rate limited. Retry after 30s.",
|
||||
"retry_after": 30
|
||||
}
|
||||
```
|
||||
|
||||
The `code` field is the stable identifier (see table above); `message` is the
|
||||
human string and may change. Some errors include extra fields
|
||||
(`retry_after`, `method_id` when `-v/--verbose` is set, etc.). Automation
|
||||
should branch on `code` (or, more simply, on the exit code).
|
||||
|
||||
**Post-parse `ClickException` is covered.** Validation failures that a
|
||||
command body or its service-layer code chooses to express by raising
|
||||
`click.UsageError` / `click.BadParameter` / `click.ClickException` (for
|
||||
example, a flag-combination conflict detected after argv parsing
|
||||
succeeds) are routed through this same envelope under `--json` and exit
|
||||
`1` with `code: "VALIDATION_ERROR"` (or another standard code chosen by
|
||||
the raise site). The contract decision and its enumerated raise sites
|
||||
are recorded in
|
||||
[ADR-0015](adr/0015-json-envelope-contract-for-post-parse-click-exceptions.md).
|
||||
|
||||
**Parse-time `ClickException` is also covered under `--json`.**
|
||||
`ClickException` raised by Click's own parser (before the command body runs)
|
||||
is JSON-wrapped at the root group (`SectionedGroup.main`), not by the
|
||||
per-command `handle_errors(...)`: under `--json` it emits the same envelope on
|
||||
stdout with `code: "VALIDATION_ERROR"` and an **exit code preserved** from
|
||||
Click (`2` for `UsageError` / `BadParameter`, `1` for the base
|
||||
`ClickException`). In text mode it is unchanged — Click renders its
|
||||
`Usage: ... / Error: ...` text on stderr and exits with that class-default
|
||||
code. This is the more consistent behavior for JSON consumers: argv-level
|
||||
failures no longer fall back to usage prose just because they fired before the
|
||||
command body. The behavior is pinned by
|
||||
[`tests/unit/cli/test_json_validation_contract.py`](../tests/unit/cli/test_json_validation_contract.py)
|
||||
(`test_json_validation_errors_emit_json`, `test_text_validation_errors_keep_click_usage_output`)
|
||||
and amended into ADR-0015 (amendment note, 2026-06-02).
|
||||
|
||||
## Intentional exceptions to the standard convention
|
||||
|
||||
Two commands deliberately extend the standard codes (or expose an opt-in
|
||||
inversion) because their primary use case is shell control flow. Code
|
||||
referencing them should comment the inverted/extended semantics. Both
|
||||
exceptions are stable contracts: `source wait`'s three-way exit (`0`/`1`/`2`)
|
||||
is by design and will not change; `source stale` follows the standard
|
||||
convention by default and only inverts when callers explicitly pass
|
||||
`--exit-on-stale`.
|
||||
|
||||
<a id="source-stale-exit-on-stale"></a>
|
||||
|
||||
### `notebooklm source stale <SOURCE_ID>` — opt-in inverted predicate
|
||||
|
||||
Implemented by `source_stale` + `_render_source_stale_result` in
|
||||
[`src/notebooklm/cli/source_cmd.py`](../src/notebooklm/cli/source_cmd.py).
|
||||
Default behavior was previously the inverted predicate (`0=stale, 1=fresh`)
|
||||
but has been standardised; the inversion is now an explicit opt-in via
|
||||
`--exit-on-stale`.
|
||||
|
||||
Default (no flag) — standard CLI convention:
|
||||
|
||||
| Exit | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Freshness check succeeded (source may be **stale** or **fresh** — branch on stdout text or, with `--json`, on the `stale`/`fresh` fields) |
|
||||
| `1` | Error (auth, network, validation, unresolvable source ID, etc. — raised by `handle_errors`) |
|
||||
|
||||
Opt-in with `--exit-on-stale` — back-compat inverted predicate:
|
||||
|
||||
| Exit | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Source is **stale** (needs `source refresh`) |
|
||||
| `1` | Source is **fresh** **or** an error occurred (ambiguous — see below) |
|
||||
|
||||
The inversion preserves the natural shell idiom for callers that depend on it:
|
||||
|
||||
```bash
|
||||
if notebooklm source stale --exit-on-stale "$SRC_ID"; then
|
||||
notebooklm source refresh "$SRC_ID"
|
||||
fi
|
||||
```
|
||||
|
||||
A `0` exit (with `--exit-on-stale`) reads as "yes, the predicate (stale)
|
||||
holds, run the body" — the same convention as `test`, `grep -q`, etc.
|
||||
|
||||
> **Important — exit-1 ambiguity (only with `--exit-on-stale`).** The
|
||||
> command is wrapped by the standard `handle_errors` context, so
|
||||
> `AuthError`, `NetworkError`, `ValidationError`, an unresolvable source
|
||||
> ID, etc. *also* exit `1` under `--exit-on-stale` and are indistinguishable
|
||||
> from "source is fresh" by exit code alone. The naive `if`-chain above
|
||||
> will silently skip the refresh body on an auth/network outage. For
|
||||
> unattended scripts, validate the session first (`notebooklm auth check --test`
|
||||
> for a live RPC probe, or `notebooklm auth check` for local cookie checks),
|
||||
> wrap with `|| die "..."` on the predicate, or
|
||||
> branch on the JSON `stale`/`fresh` fields with the default (non-opt-in)
|
||||
> semantics where success and freshness verdict are decoupled.
|
||||
|
||||
Note: under `set -e` the `1` exit (when fresh, with `--exit-on-stale`)
|
||||
will abort the script. Use the predicate inside an `if`/`elif`/`||` (as
|
||||
above), which shell's errexit explicitly excludes, or `set +e` around the
|
||||
call. The default semantics (no flag) do not have this hazard — the
|
||||
command exits `0` on success regardless of freshness.
|
||||
|
||||
<a id="source-wait-three-way"></a>
|
||||
|
||||
### `notebooklm source wait <SOURCE_ID>` — three-way
|
||||
|
||||
Implemented by `source_wait` in
|
||||
[`src/notebooklm/cli/source_cmd.py`](../src/notebooklm/cli/source_cmd.py) (the
|
||||
exit-code table is in the command's docstring, around lines 1080-1084 at
|
||||
the time of writing).
|
||||
|
||||
| Exit | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Source is ready |
|
||||
| `1` | Source not found or processing failed |
|
||||
| `2` | Timeout reached before the source became ready |
|
||||
|
||||
This is the only command whose `2` exit does **not** indicate a bug — it is
|
||||
a recoverable condition the caller may want to retry with a longer
|
||||
`--timeout`. Scripts that distinguish "transient" from "fatal" should branch
|
||||
on the specific code rather than the truthy/falsy value:
|
||||
|
||||
```bash
|
||||
notebooklm source wait "$SRC_ID" --timeout 300
|
||||
case $? in
|
||||
0) echo "ready" ;;
|
||||
1) echo "failed"; exit 1 ;;
|
||||
2) echo "timed out, retry later"; exit 75 ;; # EX_TEMPFAIL
|
||||
*) echo "unexpected"; exit 1 ;;
|
||||
esac
|
||||
```
|
||||
|
||||
## Recipes for callers
|
||||
|
||||
### Shell
|
||||
|
||||
```bash
|
||||
# Standard — non-zero is failure
|
||||
notebooklm ask -n "$NOTEBOOK_ID" "Summarize"
|
||||
status=$?
|
||||
if [ "$status" -ne 0 ]; then
|
||||
echo "ask failed (exit $status)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Distinguish bug from user error
|
||||
notebooklm <cmd> --json > out.json
|
||||
case $? in
|
||||
0) ;; # success
|
||||
1) jq -r .code out.json ;; # user/app error — branch on code
|
||||
2) echo "internal CLI error" >&2 ;; # bug; report it
|
||||
130) echo "cancelled by user" >&2 ;; # ^C
|
||||
esac
|
||||
```
|
||||
|
||||
### Python `subprocess`
|
||||
|
||||
```python
|
||||
import json
|
||||
import subprocess
|
||||
import time
|
||||
|
||||
result = subprocess.run(
|
||||
["notebooklm", "ask", "-n", nb_id, prompt, "--json"],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
payload = json.loads(result.stdout)
|
||||
elif result.returncode == 1:
|
||||
err = json.loads(result.stdout) # JSON error document
|
||||
if err["code"] == "RATE_LIMITED":
|
||||
time.sleep(err.get("retry_after", 30))
|
||||
elif result.returncode == 2:
|
||||
raise RuntimeError(f"CLI bug: {result.stdout}")
|
||||
elif result.returncode == 130:
|
||||
raise KeyboardInterrupt
|
||||
```
|
||||
|
||||
## Migration notes
|
||||
|
||||
The following shifts have landed (or are about to land) as part of the CLI
|
||||
UX overhaul and are documented here for callers preparing for — or recovering
|
||||
from — the contract change.
|
||||
|
||||
<a id="get-on-not-found-exits-1-was-0-landed"></a>
|
||||
|
||||
### `get`-on-not-found exits `1` (was `0`) ✅ **Landed**
|
||||
|
||||
`notebooklm source get`, `notebooklm artifact get`, and `notebooklm note get`
|
||||
**now exit `1`** with the typed JSON error envelope (`{error, code:
|
||||
"NOT_FOUND", message, ...}` under `--json`; plain "X not found" on stderr
|
||||
otherwise) when the requested ID is missing. Previously they printed a "not
|
||||
found" message to stdout and exited `0`. The new contract matches the rest of
|
||||
the CLI's user-error convention and lets scripts branch on the exit code
|
||||
without parsing output text:
|
||||
|
||||
```bash
|
||||
# Idiomatic
|
||||
if ! notebooklm source get "$SRC_ID"; then
|
||||
handle_missing "$SRC_ID"
|
||||
fi
|
||||
|
||||
# JSON form — branch on the typed code
|
||||
notebooklm source get "$SRC_ID" --json > out.json
|
||||
case $? in
|
||||
0) ;; # found; ``out.json`` mirrors the Source dataclass
|
||||
1) jq -r .code out.json ;; # ``NOT_FOUND`` here, but auth/network errors also land
|
||||
esac
|
||||
```
|
||||
|
||||
This **breaks** any shell script that relied on exit-`0`-on-not-found (e.g.
|
||||
`notebooklm source get X | grep -q '<title>' && do_something`). Such scripts
|
||||
must switch to the new exit-code branch shown above. The message text is also
|
||||
no longer printed to stdout (it's on stderr now), so `grep`-on-stdout for
|
||||
"not found" likewise stops working — branch on the exit code instead.
|
||||
|
||||
The change covers **both** code paths: input IDs ≥20 chars (which skip the
|
||||
partial-resolve list round-trip in `_resolve_partial_id`) and the rare
|
||||
race where partial-resolve succeeds but the subsequent `get` returns
|
||||
`None` because the row was deleted between the two calls.
|
||||
|
||||
The pre-existing "no partial-ID match" branch (raised by `_resolve_partial_id`
|
||||
as a `ClickException`) was already exit `1` and is unchanged.
|
||||
|
||||
### `note`/`artifact delete --json` without `--yes`, `note rename` race exit `1` (was `0`) ✅ **Landed**
|
||||
|
||||
These surgical fixes match the broader `--json` exit-code convention pinned
|
||||
by the prior `get`-on-not-found change. The affected cases previously emitted
|
||||
a `{"<verb>ed": false, "error": ...}` payload on exit `0`, which passed
|
||||
silently in `set -e` / `check_call`-style scripts branching on the exit code.
|
||||
|
||||
`notebooklm note delete <id> --json` without `--yes` cannot prompt (it would
|
||||
corrupt the parseable-JSON contract callers depend on), but it must also not
|
||||
appear to succeed. The command now emits the standard typed envelope:
|
||||
|
||||
```json
|
||||
{
|
||||
"error": true,
|
||||
"code": "VALIDATION_ERROR",
|
||||
"message": "Pass --yes to confirm deletion in --json mode",
|
||||
"id": "...",
|
||||
"notebook_id": "..."
|
||||
}
|
||||
```
|
||||
|
||||
on stdout and exits `1`. The text-mode interactive prompt path (no `--json`)
|
||||
is unchanged — declining the prompt is still a no-op exit `0`.
|
||||
|
||||
`notebooklm artifact delete <id> --json` follows the same destructive-operation
|
||||
guard: without `--yes`, it now emits `VALIDATION_ERROR`, includes the resolved
|
||||
artifact and notebook IDs plus `"deleted": false`, exits `1`, and does not
|
||||
delete the artifact. Passing `--yes` preserves the existing successful JSON
|
||||
payload (`{"id": "...", "deleted": true}`).
|
||||
|
||||
`notebooklm note rename <id> "new title"` resolves the partial ID and then
|
||||
fetches the current note to preserve its content before issuing the update.
|
||||
A concurrent `note delete` can win the race between those two calls; the
|
||||
backend then returns `None` and the rename has nothing to update. The
|
||||
command now funnels that race into the same typed `NOT_FOUND` envelope used
|
||||
by `note get`'s Path B:
|
||||
|
||||
```json
|
||||
{
|
||||
"error": true,
|
||||
"code": "NOT_FOUND",
|
||||
"message": "Note not found",
|
||||
"id": "...",
|
||||
"notebook_id": "..."
|
||||
}
|
||||
```
|
||||
|
||||
on stdout (text mode: plain `Note not found` on stderr) and exits `1`. The
|
||||
update RPC is **never** issued on this path — callers can rely on the
|
||||
absence of side effects when they see this envelope.
|
||||
|
||||
**Migration:** scripts branching on the exit code now correctly catch both
|
||||
misconfigurations. Scripts parsing the JSON body must switch from
|
||||
`data["deleted"] == false` / `data["renamed"] == false` to
|
||||
`data["error"] == true` (or branch on `data["code"]`).
|
||||
|
||||
### `download` exception paths route through the typed handler
|
||||
|
||||
The `download` command group routes all `download` exception paths through `handle_errors` (`cli/download_cmd.py`) so that:
|
||||
|
||||
- `--json` consistently produces the JSON error document on every failure.
|
||||
- Exit codes match the standard table above (`1` for known library errors,
|
||||
`2` for unexpected, `130` for `^C`).
|
||||
|
||||
## See also
|
||||
|
||||
- [CLI Reference](cli-reference.md) — command-by-command documentation
|
||||
- [Configuration](configuration.md) — `--json` and global options
|
||||
- [Troubleshooting](troubleshooting.md) — interpreting common errors
|
||||
- [`src/notebooklm/cli/error_handler.py`](../src/notebooklm/cli/error_handler.py)
|
||||
— canonical implementation
|
||||
|
||||
## Exit code semantics
|
||||
|
||||
This is the normative one-line summary of the convention every
|
||||
`notebooklm` CLI command obeys unless it appears in the
|
||||
[Intentional exceptions](#intentional-exceptions-to-the-standard-convention)
|
||||
section above.
|
||||
|
||||
| Code | Semantic meaning |
|
||||
|------|------------------|
|
||||
| `0` | The command succeeded as documented — the requested effect was carried out and any reported result is authoritative. |
|
||||
| `1` | The command failed, **or** the queried target was not found. Both share exit `1` because automation typically wants the same control-flow branch (`if !` / `case 1)`); JSON mode (`--json`) distinguishes them via the typed `code` field (`NOT_FOUND` vs. `AUTH_ERROR` vs. `VALIDATION_ERROR`, etc.). |
|
||||
| `2` | Click parser-time error — argv could not be parsed into a valid command invocation (unknown flag, type-validation failure, missing required argument). Under `--json` the root group still emits the typed JSON envelope on stdout (`code: "VALIDATION_ERROR"`) but **preserves** this exit `2`; in text mode Click renders its `Usage:/Error:` prose on stderr. See the [parser-time row in the Exception → exit-code mapping](#exception-exit-code-mapping) for the full behavior; this entry exists to call out that `2` is **not** a post-parse code in the default case. Post-parse `ClickException` is contracted by [ADR-0015](adr/0015-json-envelope-contract-for-post-parse-click-exceptions.md) to route through the typed JSON envelope and exit `1`, not `2`. The same code is also raised when `handle_errors` catches an unhandled non-`NotebookLMError` exception (likely a bug — see the [Standard exit codes](#standard-exit-codes) table). |
|
||||
|
||||
Two commands deliberately deviate from this baseline because their primary
|
||||
use case is shell control flow:
|
||||
|
||||
- `source wait` extends the table with `2` = timeout (a recoverable condition,
|
||||
not a bug — the only command where `2` is not a parser-time error). See
|
||||
[`notebooklm source wait`](#source-wait-three-way).
|
||||
- `source stale` offers an opt-in inverted predicate via `--exit-on-stale`
|
||||
(`0=stale, 1=fresh`) for back-compat with the `if … ; then refresh; fi`
|
||||
idiom. The default now follows the standard convention. See
|
||||
[`notebooklm source stale`](#source-stale-exit-on-stale).
|
||||
|
||||
`130` (Ctrl-C / SIGINT) is signal-driven and orthogonal to the
|
||||
success/failure axis; it is documented in the
|
||||
[Standard exit codes](#standard-exit-codes) table above.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,730 @@
|
||||
# Configuration
|
||||
|
||||
**Status:** Active
|
||||
**Last Updated:** 2026-07-04
|
||||
|
||||
This guide covers storage locations, environment settings, and configuration options for `notebooklm-py`.
|
||||
|
||||
## File Locations
|
||||
|
||||
All data is stored under `~/.notebooklm/` by default, organized by profile:
|
||||
|
||||
```
|
||||
~/.notebooklm/
|
||||
├── config.json # Global config: default_profile, language
|
||||
├── profiles/
|
||||
│ ├── default/ # Default profile (auto-created)
|
||||
│ │ ├── storage_state.json # Authentication cookies and session
|
||||
│ │ ├── context.json # CLI context (active notebook, conversation)
|
||||
│ │ └── browser_profile/ # Persistent Chromium profile
|
||||
│ ├── work/ # Named profile example
|
||||
│ │ ├── storage_state.json
|
||||
│ │ ├── context.json
|
||||
│ │ └── browser_profile/
|
||||
│ └── personal/
|
||||
│ └── ...
|
||||
```
|
||||
|
||||
`config.json` stores process-wide settings — the persisted default profile
|
||||
name (under the `default_profile` key) and the configured interface language
|
||||
(`language`). It is **global, not per-profile** (see
|
||||
`src/notebooklm/paths.py` and `src/notebooklm/cli/language_cmd.py`).
|
||||
|
||||
**Legacy layout:** If upgrading from a pre-profile version, the first run auto-migrates flat files into `profiles/default/`. The migration runs under a single-writer `filelock` rooted at `~/.notebooklm/.migration.lock`, so concurrent CLI invocations (e.g., container start-up races) cannot interleave copies — the loser of the lock re-checks the completion marker and no-ops (see `src/notebooklm/migration.py`). The legacy flat layout continues to work as a fallback.
|
||||
|
||||
You can relocate all files by setting `NOTEBOOKLM_HOME`:
|
||||
|
||||
```bash
|
||||
export NOTEBOOKLM_HOME=/custom/path
|
||||
# All files now go to /custom/path/profiles/<profile>/
|
||||
```
|
||||
|
||||
### Storage State (`storage_state.json`)
|
||||
|
||||
Contains the authentication data extracted from your browser session:
|
||||
|
||||
```json
|
||||
{
|
||||
"cookies": [
|
||||
{
|
||||
"name": "SID",
|
||||
"value": "...",
|
||||
"domain": ".google.com",
|
||||
"path": "/",
|
||||
"expires": 1234567890,
|
||||
"httpOnly": true,
|
||||
"secure": true,
|
||||
"sameSite": "Lax"
|
||||
},
|
||||
...
|
||||
],
|
||||
"origins": [],
|
||||
"notebooklm": {
|
||||
"version": 1,
|
||||
"account": {
|
||||
"authuser": 0,
|
||||
"email": "you@example.com"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Cookie requirements** (empirically validated via single- and pair-wise ablation, see `auth-cookie-lifecycle.md` §3.5; enforced by `_validate_required_cookies()` in `auth.py`):
|
||||
|
||||
- **Tier 1 — strictly required (raises on absence):** `SID` AND `__Secure-1PSIDTS`. `SID` is the only individually-required cookie (`__Secure-1PSIDTS` is removable on its own because Google can re-mint it via `RotateCookies`), but the pair-wise check uncovered that as soon as `__Secure-1PSIDTS` and any one other auth cookie are both missing, Google rejects with `Authentication expired or invalid`. The library therefore enforces both up-front. Authoritative value: `MINIMUM_REQUIRED_COOKIES` in `auth.py`.
|
||||
- **Tier 2 — secondary binding (logs a warning if absent):** either `OSID` is present, or both `APISID` and `SAPISID` are present. Without this, even valid Tier 1 cookies can't authenticate the homepage GET. Logged rather than raised so unverified edge-case flows (e.g. Workspace SSO) aren't broken by a too-strict client check.
|
||||
|
||||
In practice: extract the full cookie set via `notebooklm login` and don't try to subset it. Partial extractions (a known failure mode of browser-cookies tooling under Chrome 127+ App-Bound Encryption) are the leading suspect for "auth expires immediately" reports — see [#371](https://github.com/teng-lin/notebooklm-py/issues/371).
|
||||
|
||||
The optional `notebooklm.account` block records the Google account route for
|
||||
multi-account sessions. New `notebooklm login` runs write it when the active
|
||||
account can be discovered safely. Older Playwright-created files can be repaired
|
||||
with `notebooklm auth refresh` when the storage state maps to one visible
|
||||
account; if several accounts are visible, use
|
||||
`notebooklm login --browser-cookies <browser> --account EMAIL` to bind the
|
||||
intended account explicitly. `auth refresh` only repairs absent metadata; if
|
||||
metadata exists but points at the wrong account, re-bind explicitly with the
|
||||
browser-cookie login path.
|
||||
|
||||
**Override location:**
|
||||
```bash
|
||||
notebooklm --storage /path/to/storage_state.json list
|
||||
```
|
||||
|
||||
### Context File (`context.json`)
|
||||
|
||||
Stores the current CLI context, such as the active notebook:
|
||||
|
||||
```json
|
||||
{
|
||||
"notebook_id": "abc123def456",
|
||||
"title": "Quarterly review notes",
|
||||
"is_owner": true,
|
||||
"created_at": "2026-05-01T17:43:21Z"
|
||||
}
|
||||
```
|
||||
|
||||
Field summary:
|
||||
|
||||
- `notebook_id` — currently selected notebook, written by `notebooklm use` and read by every command that takes `-n/--notebook`.
|
||||
- `title`, `is_owner`, `created_at` — optional notebook metadata captured at selection time so `status` / display commands don't need an extra round-trip. Omitted when the CLI didn't have the values to write (see `src/notebooklm/cli/helpers.py:623-651`).
|
||||
|
||||
This file is managed automatically by `notebooklm use`, `notebooklm clear`, and the `auth` commands.
|
||||
|
||||
### Browser Profile (`browser_profile/`)
|
||||
|
||||
A persistent Chromium user data directory used during `notebooklm login`.
|
||||
|
||||
**Why persistent?** Google blocks automated login attempts. A persistent profile makes the browser appear as a regular user installation, avoiding bot detection.
|
||||
|
||||
**To reset:** Delete the `browser_profile/` directory and run `notebooklm login` again.
|
||||
|
||||
### Master Token (`master_token.json`)
|
||||
|
||||
Written only by `notebooklm login --master-token` (the `[headless]` extra). Holds
|
||||
a durable Google master token (mode `0600`) that mints/refreshes the profile's
|
||||
`storage_state.json` cookies with no per-session browser. When present beside a
|
||||
profile's `storage_state.json`, an expired session re-mints from it
|
||||
automatically.
|
||||
|
||||
```json
|
||||
{"version": 1, "email": "...", "android_id": "<hex>", "master_token": "aas_et/..."}
|
||||
```
|
||||
|
||||
> ⚠️ **Full-account, durable credential** — larger blast radius than
|
||||
> `storage_state.json`; dedicated/throwaway account only. See
|
||||
> [installation.md#alternative-master-token-auth-no-cookie-file-to-ship-survives-expiry](installation.md#d-headless-server-or-ci).
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `NOTEBOOKLM_HOME` | Base directory for all files | `~/.notebooklm` |
|
||||
| `NOTEBOOKLM_PROFILE` | Active profile name | `default` |
|
||||
| `NOTEBOOKLM_AUTH_JSON` | Inline authentication JSON (for CI/CD) | - |
|
||||
| `NOTEBOOKLM_NOTEBOOK` | Default notebook ID for commands without `-n/--notebook` | - |
|
||||
| `NOTEBOOKLM_HL` | Default interface/output language code (e.g. `en`, `ja`, `zh_Hans`) | `en` |
|
||||
| `NOTEBOOKLM_BASE_URL` | NotebookLM base URL. Constrained to `https://notebooklm.google.com` (personal) or `https://notebooklm.cloud.google.com` (enterprise) | `https://notebooklm.google.com` |
|
||||
| `NOTEBOOKLM_BL` | `bl` (build label) URL parameter for the chat streaming endpoint; override when chasing a regression tied to a specific frontend build snapshot | built-in default in `_env.DEFAULT_BL` |
|
||||
| `NOTEBOOKLM_TRANSPORT` | HTTP transport backend: `httpx` (default) or `curl_cffi` (opt-in browser-TLS impersonation; requires the `curl_cffi` package). Use `curl_cffi` where the default transport is TLS-fingerprint-blocked. | `httpx` |
|
||||
| `NOTEBOOKLM_LOG_LEVEL` | Logging level: `DEBUG`, `INFO`, `WARNING`, `ERROR` | `WARNING` |
|
||||
| `NOTEBOOKLM_DEBUG_RPC` | Legacy: Enable RPC debug logging (use `LOG_LEVEL=DEBUG` instead) | `false` |
|
||||
| `NOTEBOOKLM_DEBUG` | Show untruncated RPC response bodies in error messages instead of the default 80-char preview (verbose; intended for deep debugging) | `0` |
|
||||
| `NOTEBOOKLM_STRICT_DECODE` | **Retired (ignored since v0.7.0).** Strict decoding is now the only mode: `safe_index` always raises `UnknownRPCMethodError` on schema drift. The former `0` warn-and-fallback opt-out was removed. | (ignored) |
|
||||
| `NOTEBOOKLM_RPC_OVERRIDES` | JSON object mapping `RPCMethod` enum names to RPC ID strings (community self-patch when Google rotates a method ID; e.g. `{"LIST_NOTEBOOKS":"AbC123"}`) | - |
|
||||
| `NOTEBOOKLM_REFRESH_CMD` | Optional command (argv list, or shell string with `_USE_SHELL=1`) invoked when auth refresh is required. Must exit `0` after writing a refreshed `storage_state.json`; the parent reloads from disk | - |
|
||||
| `NOTEBOOKLM_REFRESH_CMD_USE_SHELL` | Opt the `NOTEBOOKLM_REFRESH_CMD` subprocess back into `shell=True` execution. Default `shell=False` (argv list) — set to the literal `1` (only `"1"` is honored — not `true`/`yes`/`on`) when the refresh command requires shell metacharacters | `0` |
|
||||
| `NOTEBOOKLM_REFRESH_PROFILE` | Child-process hint set for `NOTEBOOKLM_REFRESH_CMD`; names the resolved profile being refreshed | resolved profile |
|
||||
| `NOTEBOOKLM_REFRESH_STORAGE_PATH` | Child-process hint set for `NOTEBOOKLM_REFRESH_CMD`; path to the `storage_state.json` file the command must rewrite | resolved storage path |
|
||||
| `NOTEBOOKLM_DISABLE_KEEPALIVE_POKE` | Disable the proactive `accounts.google.com/RotateCookies` poke that refreshes `__Secure-1PSIDTS` ahead of expiry | `0` |
|
||||
| `NOTEBOOKLM_HEADLESS_REAUTH` | Opt in to layer-3 headless re-auth during automatic refresh paths. Explicit `client.refresh_auth(allow_headless=True)` does not require this env var. | `0` |
|
||||
| `NOTEBOOKLM_HEADLESS_REAUTH_CDP_URL` | Optional loopback Chrome DevTools endpoint for layer-3 headless re-auth, e.g. `http://127.0.0.1:9222`. Non-loopback endpoints are ignored for credential safety. | - |
|
||||
| `NOTEBOOKLM_MCP_TRANSPORT` | MCP server transport for `notebooklm-mcp`: `stdio` or `http` | `stdio` |
|
||||
| `NOTEBOOKLM_MCP_HOST` | MCP HTTP transport bind host; non-loopback refused unless `NOTEBOOKLM_MCP_ALLOW_EXTERNAL_BIND=1` | `127.0.0.1` |
|
||||
| `NOTEBOOKLM_MCP_PORT` | MCP HTTP transport bind port | `9420` |
|
||||
| `NOTEBOOKLM_MCP_ALLOW_EXTERNAL_BIND` | Allow MCP HTTP transport to bind a non-loopback host. Use only behind a trusted proxy. | `0` |
|
||||
| `NOTEBOOKLM_MCP_OAUTH_PASSWORD` | Password gating the self-hosted OAuth authorization server that lets claude.ai connect to the remote MCP server (≥16 chars). Set together with `NOTEBOOKLM_MCP_OAUTH_BASE_URL`; both unset → bearer-only. | - |
|
||||
| `NOTEBOOKLM_MCP_OAUTH_BASE_URL` | Bare public HTTPS origin (no path) the self-hosted OAuth endpoints (`/authorize`, `/token`, `/.well-known/*`) mount under. Required with `NOTEBOOKLM_MCP_OAUTH_PASSWORD`; partial/weak/non-HTTPS config refuses to start. | - |
|
||||
| `NOTEBOOKLM_MCP_PUBLIC_URL` | Public base URL for the remote MCP file upload/download signed-URL side-channel (falls back to `NOTEBOOKLM_MCP_OAUTH_BASE_URL`). Unset → `source_add type=file` / `artifact_download` return a "not configured" error. | - |
|
||||
| `NOTEBOOKLM_MCP_TRUST_PROXY` | Trust the proxy-set `CF-Connecting-IP` header as the self-hosted-OAuth login-throttle key. Only enable behind a trusted proxy (e.g. the Cloudflare tunnel); default off keys on the socket peer. | `0` |
|
||||
| `NOTEBOOKLM_MCP_STRICT_IDS` | Strict IDs-only mode for MCP tools: require a full canonical id for every `notebook`/`source`/`note`/`artifact` reference and reject names, titles, and short id prefixes (fail-fast, deterministic automation). | `0` |
|
||||
| `NOTEBOOKLM_SERVER_TOKEN` | Bearer token required by every REST `/v1` request. The REST server refuses to start without it. | - |
|
||||
| `NOTEBOOKLM_SERVER_HOST` | REST server bind host; non-loopback refused unless `NOTEBOOKLM_SERVER_ALLOW_EXTERNAL_BIND=1` | `127.0.0.1` |
|
||||
| `NOTEBOOKLM_SERVER_PORT` | REST server bind port | `8000` |
|
||||
| `NOTEBOOKLM_SERVER_ALLOW_EXTERNAL_BIND` | Allow REST server to bind a non-loopback host. Use only behind a trusted proxy. | `0` |
|
||||
| `NOTEBOOKLM_SERVER_SOURCE_MUTATION_CONCURRENCY` | Max concurrent REST source create/rename/delete/batch handlers. | `4` |
|
||||
| `NOTEBOOKLM_SERVER_SOURCE_WAIT_CONCURRENCY` | Max concurrent REST source wait handlers. | `4` |
|
||||
| `NOTEBOOKLM_SERVER_GENERATION_CONCURRENCY` | Max concurrent REST artifact generation/retry handlers. | `2` |
|
||||
| `NOTEBOOKLM_SERVER_DOWNLOAD_CONCURRENCY` | Max concurrent REST artifact download handlers. | `2` |
|
||||
| `NOTEBOOKLM_SERVER_RESEARCH_CONCURRENCY` | Max concurrent REST research start/cancel/import handlers. | `2` |
|
||||
| `NOTEBOOKLM_SERVER_CHAT_CONCURRENCY` | Max concurrent REST blocking chat ask handlers. | `4` |
|
||||
| `NOTEBOOKLM_QUIET_DEPRECATIONS` | Suppress the project's public-API `DeprecationWarning`s (the one-off warnings routed through `warn_deprecated`, e.g. awaiting `from_storage(...)`). Set to a truthy value (`1` / `true` / `yes` / `on`, case-insensitive) to silence them; see `docs/deprecations.md`. | (warnings emitted) |
|
||||
| `NOTEBOOKLM_FUTURE_ERRORS` | **Retired (removed in v0.8.0; ignored).** It was the v0.7.0 forward-compat preview gate for the v0.8.0 error contract; now that every break it staged is the default, the flag is a no-op — setting it has no effect. See `docs/deprecations.md`. | (ignored) |
|
||||
| `NOTEBOOKLM_VCR_RECORD_ERRORS` | Synthetic-error injection mode for VCR test cassettes (`429`, `5xx`, `expired_csrf`) | - |
|
||||
|
||||
### Public config API vs internal resolvers
|
||||
|
||||
`src/notebooklm/_env.py` owns internal environment/default resolution for
|
||||
runtime behavior. It reads process environment variables and contains internal
|
||||
defaults such as `DEFAULT_BL` for the chat streaming build label.
|
||||
|
||||
`notebooklm.config` is the stable public import surface. It intentionally
|
||||
re-exports only the supported endpoint/language helpers:
|
||||
`DEFAULT_BASE_URL`, `ENTERPRISE_BASE_HOST`, `get_base_host`, `get_base_url`,
|
||||
`get_default_language`, and `PERSONAL_BASE_HOST`. Existing imports
|
||||
from `notebooklm.config` remain supported; internal-only `_env` names should
|
||||
not be imported by downstream code.
|
||||
|
||||
### Env vars and precedence
|
||||
|
||||
Every `NOTEBOOKLM_*` variable read by the library and CLI, in one place. CLI
|
||||
flags always win over env vars; env vars win over persisted profile config /
|
||||
context; built-in defaults are the last fallback. The "Resolved by" column
|
||||
points at the canonical resolver so the precedence rule for each variable can
|
||||
be audited from one location.
|
||||
|
||||
| Variable | Purpose | Resolution order (highest → lowest) | Resolved by |
|
||||
|----------|---------|-------------------------------------|-------------|
|
||||
| `NOTEBOOKLM_PROFILE` | Active profile name. Selects which `~/.notebooklm/profiles/<name>/` directory backs storage and context. | `-p/--profile` flag → `NOTEBOOKLM_PROFILE` → `default_profile` from `~/.notebooklm/config.json` → `default` | `paths.resolve_profile` |
|
||||
| `NOTEBOOKLM_AUTH_JSON` | Inline `storage_state.json` payload for CI/CD; bypasses on-disk profile storage entirely. | `--storage` flag → `NOTEBOOKLM_AUTH_JSON` → profile-aware `storage_state.json` → legacy fallback | `auth.load_auth_from_storage` |
|
||||
| `NOTEBOOKLM_HOME` | Base directory for all per-profile files. | `NOTEBOOKLM_HOME` → `~/.notebooklm` | `paths.get_home_dir` |
|
||||
| `NOTEBOOKLM_HL` | Default interface/output language for `generate <kind>` and the `hl` query parameter on every batchexecute RPC. | `--language` flag → `NOTEBOOKLM_HL` → `language` value from **global** `~/.notebooklm/config.json` (NOT per-profile) → `en` | `language.resolve_hl` |
|
||||
| `NOTEBOOKLM_LOG_LEVEL` | `DEBUG`/`INFO`/`WARNING`/`ERROR` floor for the `notebooklm` package logger. | `--quiet` flag (forces `ERROR`) → `-v/-vv` flags (force `INFO`/`DEBUG`) → `NOTEBOOKLM_DEBUG_RPC=1` (forces `DEBUG`) → `NOTEBOOKLM_LOG_LEVEL` → `WARNING` | `_logging.configure_logging` + `notebooklm_cli.cli` |
|
||||
| `NOTEBOOKLM_DEBUG_RPC` | Legacy alias that sets the package logger to `DEBUG`. Prefer `NOTEBOOKLM_LOG_LEVEL=DEBUG` for new code. | (See `NOTEBOOKLM_LOG_LEVEL`.) | `_logging.configure_logging` |
|
||||
| `NOTEBOOKLM_NOTEBOOK` | Default notebook ID when no `-n/--notebook` flag is passed. Composes with `notebooklm use <id>` so per-shell overrides do not clobber the persisted active-notebook context. | `-n/--notebook` flag → `NOTEBOOKLM_NOTEBOOK` → active context (from `notebooklm use`) → error | `cli.helpers.require_notebook` (Click also reads it natively via `cli/options.py:notebook_option`'s `envvar=`) |
|
||||
| `NOTEBOOKLM_RPC_OVERRIDES` | **JSON object** mapping `RPCMethod` enum names to RPC ID strings (e.g. `{"LIST_NOTEBOOKS": "AbC123"}`). Overrides runtime RPC IDs — community self-patch when Google rotates a method ID. Empty string / unset disables the mechanism; invalid JSON or non-object payloads emit a `WARNING` and are ignored. | Process env, evaluated per RPC resolve (cached on the raw env string). | `notebooklm.rpc.overrides._parse_rpc_overrides` |
|
||||
| `NOTEBOOKLM_QUIET_DEPRECATIONS` | Suppress the project's public-API `DeprecationWarning`s — the one-off warnings routed through `src/notebooklm/_deprecation.py::warn_deprecated` (e.g. awaiting `from_storage(...)`). Set to a truthy value (`1` / `true` / `yes` / `on`) to silence them. See `docs/deprecations.md`. | (warnings emitted) | `_deprecation._deprecations_quiet` / `deprecations_quiet` |
|
||||
| `NOTEBOOKLM_FUTURE_ERRORS` | **Retired (removed in v0.8.0; ignored).** It was the v0.7.0 forward-compat preview gate for the v0.8.0 error contract (ADR-0019 / umbrella [#1346](https://github.com/teng-lin/notebooklm-py/issues/1346)). Now that every break it staged — `get()` raising `*NotFoundError`, the attribute-only typed returns, the removed `interval=` alias, the bool→`None` returns, the refusal-raises, and the mutate-existing fail-loud — is the default, the flag is a **no-op**: setting it has no effect. See `docs/deprecations.md`. | (ignored) | — |
|
||||
| `NOTEBOOKLM_STRICT_DECODE` | **Retired (ignored since v0.7.0).** Strict decoding is the only mode — `safe_index` always raises `UnknownRPCMethodError` on schema drift. The former `0` warn-and-fallback opt-out was removed; setting the variable has no effect. | (ignored) | — |
|
||||
| `NOTEBOOKLM_BASE_URL` | NotebookLM base URL. Constrained to `https://notebooklm.google.com` (personal) or `https://notebooklm.cloud.google.com` (enterprise); other schemes/hosts/paths raise `ValueError`. | Process env on every base-URL lookup. | `_env.get_base_url` |
|
||||
| `NOTEBOOKLM_BL` | `bl` (build label) URL parameter sent on the chat streaming endpoint (`ChatAPI.ask`). Pins the frontend build the request is attributed to. | Process env on every chat stream call; whitespace-only falls back to `_env.DEFAULT_BL`. | `_env.get_default_bl` |
|
||||
| `NOTEBOOKLM_DEBUG` | When `1`, RPC error messages include the **full** untruncated response body instead of the default 80-char preview. Verbose; intended for deep debugging only. | Process env on each error formatting call. | `exceptions._truncate_response_preview` |
|
||||
| `NOTEBOOKLM_REFRESH_CMD` | Optional command invoked when auth refresh is required. Must exit `0` after writing a refreshed `storage_state.json`; the parent reloads cookies from disk. Stdout/stderr are not parsed (only surfaced in the non-zero-exit error message). Parsing honors `NOTEBOOKLM_REFRESH_CMD_USE_SHELL`. | Process env on each refresh subprocess spawn. | `auth` refresh-spawn helper (constant `NOTEBOOKLM_REFRESH_CMD_ENV` in `notebooklm.auth`) |
|
||||
| `NOTEBOOKLM_REFRESH_CMD_USE_SHELL` | Opt the optional `NOTEBOOKLM_REFRESH_CMD` subprocess back into `shell=True`. Default `shell=False` parses the command with `shlex.split` and invokes it as an argv list (safer; resists shell-injection footguns when the env var is sourced from CI configs or container env files). | Process env on each refresh subprocess spawn. | `auth` refresh-spawn helper (constant `NOTEBOOKLM_REFRESH_CMD_USE_SHELL_ENV` in `notebooklm.auth`) |
|
||||
| `NOTEBOOKLM_REFRESH_PROFILE` | Child env var injected into `NOTEBOOKLM_REFRESH_CMD`; names the resolved NotebookLM profile that is being refreshed. Refresh scripts may read it, but setting it in the parent shell does not select the profile. | Set by `auth` refresh-spawn helper from the resolved profile. | `auth._run_refresh_cmd` |
|
||||
| `NOTEBOOKLM_REFRESH_STORAGE_PATH` | Child env var injected into `NOTEBOOKLM_REFRESH_CMD`; points to the `storage_state.json` file the command must rewrite before exiting `0`. Refresh scripts may read it, but setting it in the parent shell does not select storage. | Set by `auth` refresh-spawn helper from the explicit storage path or profile-aware storage path. | `auth._run_refresh_cmd` |
|
||||
| `NOTEBOOKLM_DISABLE_KEEPALIVE_POKE` | When `1`, disable the proactive `accounts.google.com/RotateCookies` poke that refreshes `__Secure-1PSIDTS` ahead of expiry. Useful when running behind a proxy that rejects the extra request, or in offline test fixtures. | Process env on every keepalive check. | `auth` keepalive guards (constant `NOTEBOOKLM_DISABLE_KEEPALIVE_POKE_ENV` in `notebooklm.auth`) |
|
||||
| `NOTEBOOKLM_HEADLESS_REAUTH` | Opt in to layer-3 headless re-auth for automatic refresh paths. `client.refresh_auth(allow_headless=True)` is the explicit Python API opt-in and does not require the env var. | Literal `1` enables; all other values disabled. | `_auth.headless_reauth.headless_reauth_env_enabled` |
|
||||
| `NOTEBOOKLM_HEADLESS_REAUTH_CDP_URL` | Optional Chrome DevTools Protocol endpoint for layer-3 headless re-auth. Must be loopback (`127.0.0.1`, `::1`, or `localhost`); remote endpoints are ignored because CDP is account-equivalent. | Explicit function argument → env var → no CDP arm. | `_auth.headless_reauth.resolve_cdp_url` |
|
||||
| `NOTEBOOKLM_MCP_TRANSPORT` | Default transport for `notebooklm-mcp`: `stdio` or `http`. CLI `--transport` wins. | `--transport` flag → env var → `stdio` | `mcp.__main__._build_parser` |
|
||||
| `NOTEBOOKLM_MCP_HOST` | HTTP bind host for `notebooklm-mcp --transport http`. Non-loopback refused unless `NOTEBOOKLM_MCP_ALLOW_EXTERNAL_BIND=1`. | `--host` flag → env var → `127.0.0.1` | `mcp.__main__._build_parser` / `_serving.check_bind_allowed` |
|
||||
| `NOTEBOOKLM_MCP_PORT` | HTTP bind port for `notebooklm-mcp --transport http`. | `--port` flag → env var → `9420` | `mcp.__main__._build_parser` / `_resolve_port` |
|
||||
| `NOTEBOOKLM_MCP_ALLOW_EXTERNAL_BIND` | Allow MCP HTTP transport to bind a non-loopback host. Use only behind a trusted proxy. | Literal `1` enables; all other values disabled. | `mcp.__main__._check_http_bind_allowed` → `_serving.check_bind_allowed` |
|
||||
| `NOTEBOOKLM_MCP_TRUST_PROXY` | Trust the proxy-set `CF-Connecting-IP` header as the self-hosted-OAuth login-throttle key. Enable only behind a trusted proxy (e.g. the Cloudflare tunnel); default off keys the throttle on the socket peer. | Literal `1` enables; all other values disabled. | `mcp._oauth.get_oauth_config` / `_client_ip` |
|
||||
| `NOTEBOOKLM_MCP_STRICT_IDS` | Strict IDs-only mode: MCP `notebook`/`source`/`note`/`artifact` references must be a full canonical id; names, titles, and short id prefixes are rejected before any list call (deterministic automation). Off by default → default name/prefix/title resolution is unchanged. | Literal `1` enables; all other values disabled. | `mcp._resolve._strict_ids_enabled` |
|
||||
| `NOTEBOOKLM_SERVER_TOKEN` | Bearer token required by every REST `/v1` request. The server refuses to start when unset/empty. | `--token` flag → env var → startup failure | `server.__main__._check_token_configured` / `server._auth.require_auth` |
|
||||
| `NOTEBOOKLM_SERVER_HOST` | REST server bind host. Non-loopback refused unless `NOTEBOOKLM_SERVER_ALLOW_EXTERNAL_BIND=1`. | `--host` flag → env var → `127.0.0.1` | `server.__main__._build_parser` / `_serving.check_bind_allowed` |
|
||||
| `NOTEBOOKLM_SERVER_PORT` | REST server bind port. | `--port` flag → env var → `8000` | `server.__main__._build_parser` / `_resolve_port` |
|
||||
| `NOTEBOOKLM_SERVER_ALLOW_EXTERNAL_BIND` | Allow REST server to bind a non-loopback host. Use only behind a trusted proxy. | Literal `1` enables; all other values disabled. | `server.__main__._check_bind_allowed` → `_serving.check_bind_allowed` |
|
||||
| `NOTEBOOKLM_SERVER_SOURCE_MUTATION_CONCURRENCY` | Max concurrent REST source create/rename/delete/batch handlers. | Env var → `4`; blank uses default; must be integer `>=1`. | `server._limits.ServerLimiters.from_env` |
|
||||
| `NOTEBOOKLM_SERVER_SOURCE_WAIT_CONCURRENCY` | Max concurrent REST source wait handlers. | Env var → `4`; blank uses default; must be integer `>=1`. | `server._limits.ServerLimiters.from_env` |
|
||||
| `NOTEBOOKLM_SERVER_GENERATION_CONCURRENCY` | Max concurrent REST artifact generation/retry handlers. | Env var → `2`; blank uses default; must be integer `>=1`. | `server._limits.ServerLimiters.from_env` |
|
||||
| `NOTEBOOKLM_SERVER_DOWNLOAD_CONCURRENCY` | Max concurrent REST artifact download handlers. | Env var → `2`; blank uses default; must be integer `>=1`. | `server._limits.ServerLimiters.from_env` |
|
||||
| `NOTEBOOKLM_SERVER_RESEARCH_CONCURRENCY` | Max concurrent REST research start/cancel/import handlers. | Env var → `2`; blank uses default; must be integer `>=1`. | `server._limits.ServerLimiters.from_env` |
|
||||
| `NOTEBOOKLM_SERVER_CHAT_CONCURRENCY` | Max concurrent REST blocking chat ask handlers. | Env var → `4`; blank uses default; must be integer `>=1`. | `server._limits.ServerLimiters.from_env` |
|
||||
| `NOTEBOOKLM_VCR_RECORD_ERRORS` | Synthetic-error injection mode for VCR test cassettes. Lowercase-normalized; valid values are `429` (rate limit), `5xx` (server error), or `expired_csrf` (CSRF token expiration). Used to record synthetic error cassettes under VCR. | Process env on each request, evaluated by `ErrorInjectionMiddleware` to intercept and synthesize failures. | `_error_injection._get_error_injection_mode` |
|
||||
|
||||
**Boolean handling.** `NOTEBOOKLM_DEBUG_RPC` treats `1` / `true` / `yes`
|
||||
(case-insensitive) as truthy; everything else is falsy.
|
||||
`NOTEBOOKLM_STRICT_DECODE` is ignored as of v0.7.0 (strict decoding is the
|
||||
only mode); no value changes decoder behavior.
|
||||
`NOTEBOOKLM_QUIET_DEPRECATIONS` treats `1` / `true` / `yes` / `on`
|
||||
(case-insensitive) as truthy; any other value (including unset) leaves
|
||||
deprecation warnings enabled. It silences the project's public-API
|
||||
`DeprecationWarning`s — the `get()`-returns-`None` warning and deprecated
|
||||
keyword aliases (e.g. `ResearchAPI.wait_for_completion`'s legacy `interval=`) —
|
||||
without changing their behavior.
|
||||
`NOTEBOOKLM_NOTEBOOK` is treated as unset when empty or whitespace-only so a
|
||||
bare `export NOTEBOOKLM_NOTEBOOK=` does not block `notebooklm use` /
|
||||
`-n/--notebook` from resolving.
|
||||
|
||||
**The `--quiet` global flag.** `notebooklm --quiet <subcommand>` suppresses
|
||||
status prose and raises the `notebooklm` package logger floor to `ERROR` for
|
||||
the duration of one invocation, so cron and CI logs stay clean while real
|
||||
failures still surface. Structured `--json` payloads are still emitted. It is
|
||||
mutually exclusive with `-v/-vv` — combining the two raises a `UsageError`
|
||||
(exit `2`) since the resolved log levels conflict (`ERROR` vs `INFO`/`DEBUG`).
|
||||
For shell-wide / always-on log suppression, use `NOTEBOOKLM_LOG_LEVEL`.
|
||||
|
||||
### NOTEBOOKLM_HOME
|
||||
|
||||
Relocates all configuration files to a custom directory:
|
||||
|
||||
```bash
|
||||
export NOTEBOOKLM_HOME=/custom/path
|
||||
|
||||
# All files now go here:
|
||||
# /custom/path/profiles/<profile>/storage_state.json
|
||||
# /custom/path/profiles/<profile>/context.json
|
||||
# /custom/path/profiles/<profile>/browser_profile/
|
||||
```
|
||||
|
||||
**Use cases:**
|
||||
- Per-project isolation
|
||||
- Custom storage locations
|
||||
|
||||
### NOTEBOOKLM_PROFILE
|
||||
|
||||
Selects the active profile without changing the persisted default:
|
||||
|
||||
```bash
|
||||
export NOTEBOOKLM_PROFILE=work
|
||||
notebooklm list # Uses ~/.notebooklm/profiles/work/
|
||||
```
|
||||
|
||||
Equivalent to passing `-p work` on every command. The CLI flag takes precedence over the env var.
|
||||
|
||||
The **persisted default** is read from the `default_profile` key of
|
||||
`~/.notebooklm/config.json` (set via `notebooklm profile switch <name>`). When
|
||||
neither a `-p/--profile` flag nor `NOTEBOOKLM_PROFILE` is set, `paths.resolve_profile`
|
||||
falls back to this value (and finally to `"default"` if `config.json` doesn't
|
||||
exist or has no `default_profile` key).
|
||||
|
||||
### NOTEBOOKLM_AUTH_JSON
|
||||
|
||||
Provides authentication inline without writing files. Ideal for CI/CD:
|
||||
|
||||
```bash
|
||||
export NOTEBOOKLM_AUTH_JSON='{"cookies":[...]}'
|
||||
notebooklm list # Works without any file on disk
|
||||
```
|
||||
|
||||
**Precedence:**
|
||||
1. `--storage` CLI flag (highest)
|
||||
2. `NOTEBOOKLM_AUTH_JSON` environment variable
|
||||
3. Profile-aware path: `$NOTEBOOKLM_HOME/profiles/<profile>/storage_state.json`
|
||||
4. `~/.notebooklm/profiles/default/storage_state.json` (default)
|
||||
5. `~/.notebooklm/storage_state.json` (legacy fallback)
|
||||
|
||||
**Note:** Cannot run `notebooklm login` when `NOTEBOOKLM_AUTH_JSON` is set.
|
||||
|
||||
### `NOTEBOOKLM_REFRESH_CMD`
|
||||
|
||||
Optional. If set, this command is invoked when an auth refresh is required —
|
||||
replacing the default browser-cookie path. The contract is **exit-code based**:
|
||||
|
||||
1. The command must exit `0`.
|
||||
2. On exit, it must have written a refreshed `storage_state.json` at the
|
||||
path in `NOTEBOOKLM_REFRESH_STORAGE_PATH`. The parent sets this child
|
||||
env var to the explicit storage path or the resolved profile-aware
|
||||
storage path before spawning the command.
|
||||
|
||||
The parent then reloads cookies from disk and retries the token fetch. Stdout
|
||||
and stderr are **not** parsed — they are only captured for inclusion in the
|
||||
error message when the command exits non-zero. Read by the
|
||||
`NOTEBOOKLM_REFRESH_CMD_ENV` constant in `notebooklm.auth`.
|
||||
The child process also receives `NOTEBOOKLM_REFRESH_PROFILE` with the resolved
|
||||
profile name.
|
||||
|
||||
See also `NOTEBOOKLM_REFRESH_CMD_USE_SHELL` to opt back into `shell=True`
|
||||
parsing.
|
||||
|
||||
### NOTEBOOKLM_HL
|
||||
|
||||
Sets the default interface/output language used by the client. The value is
|
||||
passed as the `hl` query parameter on every batchexecute RPC call and is used by
|
||||
the `generate audio|video|slide-deck|infographic|data-table|mind-map|report`
|
||||
commands. In the Python API, omitted `language` on `ArtifactsAPI.generate_*`
|
||||
keeps the historical `"en"` artifact default; pass `language=None` to opt in to
|
||||
this `NOTEBOOKLM_HL` resolver.
|
||||
|
||||
```bash
|
||||
export NOTEBOOKLM_HL=ja
|
||||
notebooklm generate audio "deep dive" # Japanese audio overview
|
||||
```
|
||||
|
||||
Surrounding whitespace is stripped; an empty or whitespace-only value falls
|
||||
back to `en`. For the generate commands, the resolution order is:
|
||||
|
||||
1. `--language` CLI flag
|
||||
2. `NOTEBOOKLM_HL` environment variable
|
||||
3. `language` value from the **global** `~/.notebooklm/config.json` (set via
|
||||
`notebooklm language set <code>`). The language is stored once per
|
||||
`NOTEBOOKLM_HOME`, **not** per profile — switching `notebooklm -p work`
|
||||
does not switch the configured language. See
|
||||
`src/notebooklm/cli/language_cmd.py` for the resolver and
|
||||
`src/notebooklm/paths.py` for the storage location.
|
||||
4. `en` (built-in default)
|
||||
|
||||
### NOTEBOOKLM_QUIET_DEPRECATIONS
|
||||
|
||||
Suppresses the project's public-API `DeprecationWarning`s while you migrate. Set
|
||||
it to a truthy value (`1`, `true`, `yes`, or `on`, case-insensitive) to silence
|
||||
them; any other value (including unset) leaves them enabled.
|
||||
|
||||
It gates the one-off deprecation warnings routed through
|
||||
`src/notebooklm/_deprecation.py::warn_deprecated` — e.g. awaiting
|
||||
`NotebookLMClient.from_storage(...)` instead of using the `async with` form. (The
|
||||
v0.7.0 error-contract runways it also gated — the `get()`-returns-`None` warning,
|
||||
the `wait_for_completion(interval=...)` alias, and the dict-subscript bridge — all
|
||||
**completed their removal in v0.8.0**, so those warnings no longer exist; see
|
||||
[`deprecations.md`](deprecations.md).)
|
||||
|
||||
The helper that reads this variable is
|
||||
`notebooklm._deprecation._deprecations_quiet` (public alias
|
||||
`deprecations_quiet`).
|
||||
|
||||
```bash
|
||||
# Silence the project's public-API deprecation warnings while migrating
|
||||
export NOTEBOOKLM_QUIET_DEPRECATIONS=1
|
||||
```
|
||||
|
||||
> Note: this variable does **not** affect `source add --mime-type` /
|
||||
> `client.sources.add_file(mime_type=...)` — `mime_type` is a supported
|
||||
> parameter and emits no warning.
|
||||
|
||||
### NOTEBOOKLM_FUTURE_ERRORS (removed in v0.8.0)
|
||||
|
||||
**Removed.** This was the v0.7.0 forward-compat preview gate for the v0.8.0 error
|
||||
contract (ADR-0019 / umbrella
|
||||
[#1346](https://github.com/teng-lin/notebooklm-py/issues/1346)): setting it made
|
||||
the v0.7.0 warn-runways adopt their v0.8.0 raise-target early so you could test
|
||||
forward-compatibility before the breaking flips shipped. v0.8.0 makes every one
|
||||
of those flips the default — `get()` raising `*NotFoundError` on a miss, the
|
||||
attribute-only typed returns, the removed `interval=` alias, the bool→`None`
|
||||
returns, the refusal-raises, and the mutate-existing fail-loud — so the flag and
|
||||
its resolver were deleted. **Setting `NOTEBOOKLM_FUTURE_ERRORS` now has no
|
||||
effect.** Remove it from your environment / CI config. See
|
||||
[`deprecations.md`](deprecations.md) for the full Removed-in-v0.8.0 table.
|
||||
|
||||
### Timeouts
|
||||
|
||||
Most batchexecute RPCs issued by the client (whether through `NotebookLMClient`
|
||||
or any of the CLI commands) use a **30-second** HTTP request timeout by default,
|
||||
with a tighter **10-second** connection-establishment timeout. The shorter
|
||||
connect timeout helps surface network-level issues quickly while the read
|
||||
timeout accommodates slow server responses. The timeout is exposed as a
|
||||
constructor argument on `NotebookLMClient` (`timeout=`) for callers that need to
|
||||
tune it per-workload — see the `DEFAULT_TIMEOUT` / `DEFAULT_CONNECT_TIMEOUT`
|
||||
constants in `src/notebooklm/_runtime/config.py`.
|
||||
|
||||
The chat streaming endpoint (`ChatAPI.ask`) also exposes a separate per-read
|
||||
silence window (`chat_timeout=`). It defaults to **180 seconds** because shared
|
||||
notebooks can be slow to send the first streamed chat byte; fast metadata RPCs
|
||||
stay on the normal **30-second** timeout. A chat read timeout means the server
|
||||
sent no stream bytes for that window, either before the first byte or between
|
||||
chunks; it does not mean total generation time exceeded 30 seconds. Pass
|
||||
`chat_timeout=None` to inherit the normal client timeout for chat. The CLI
|
||||
`ask --request-timeout N` flag overrides both values for that invocation.
|
||||
|
||||
### Decoder strictness
|
||||
|
||||
NotebookLM's batchexecute responses are obfuscated, undocumented, and reshaped
|
||||
by Google without notice. The decoder uses a shared `safe_index` helper to walk
|
||||
nested response payloads. When it can't descend (an index is out of range, or
|
||||
the value at a step isn't indexable), it **raises**
|
||||
`UnknownRPCMethodError` (a subclass of `DecodingError` / `RPCError`) with
|
||||
structured `method_id`, `path`, `source`, and `data_at_failure` attributes.
|
||||
|
||||
Strict decoding is the only mode. The legacy `NOTEBOOKLM_STRICT_DECODE=0`
|
||||
warn-and-return-`None` opt-out (which emitted a `DeprecationWarning` through
|
||||
v0.5.0/v0.6.0) was **retired in v0.7.0**; the env var is now ignored. The
|
||||
strict contract surfaces real schema drift (Google rotating a response shape)
|
||||
as a typed exception instead of a silent `None` return, so callers that
|
||||
previously treated `None` as a sentinel must handle `UnknownRPCMethodError`.
|
||||
|
||||
The same `UnknownRPCMethodError` is also raised by `decode_response()` when the
|
||||
batchexecute response contains RPC IDs but not the one the call requested
|
||||
(typically a sign that Google rotated the method ID).
|
||||
|
||||
> Background and rationale for the flip: see
|
||||
> [`docs/adr/0011-schema-validation-policy.md`](adr/0011-schema-validation-policy.md).
|
||||
|
||||
## CLI Options
|
||||
|
||||
### Global Options
|
||||
|
||||
| Option | Description | Default |
|
||||
|--------|-------------|---------|
|
||||
| `--storage PATH` | Path to storage_state.json | `$NOTEBOOKLM_HOME/profiles/<profile>/storage_state.json` |
|
||||
| `-p, --profile NAME` | Use a named profile | Active profile or `default` |
|
||||
| `-v, --verbose` | Enable verbose output (`-v` for INFO, `-vv` for DEBUG) | - |
|
||||
| `--quiet` | Suppress status output and INFO/WARN logs (only errors survive). Mutually exclusive with `-v`. | - |
|
||||
| `--version` | Show version | - |
|
||||
| `--help` | Show help | - |
|
||||
|
||||
### Viewing Configuration
|
||||
|
||||
See where your configuration files are located:
|
||||
|
||||
```bash
|
||||
notebooklm status --paths
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
Configuration Paths
|
||||
┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
|
||||
┃ File ┃ Path ┃ Source ┃
|
||||
┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
|
||||
│ Profile │ default │ active │
|
||||
│ Home Directory │ /home/user/.notebooklm │ default │
|
||||
│ Storage State │ .../profiles/default/storage_state.json │ │
|
||||
│ Context │ .../profiles/default/context.json │ │
|
||||
│ Browser Profile │ .../profiles/default/browser_profile │ │
|
||||
└─────────────────┴──────────────────────────────────────────┴───────────┘
|
||||
```
|
||||
|
||||
## Session Management
|
||||
|
||||
### Session Lifetime
|
||||
|
||||
Authentication sessions are tied to Google's cookie expiration:
|
||||
- Sessions typically last several days to weeks
|
||||
- Google may invalidate sessions for security reasons
|
||||
- Rate limiting or suspicious activity can trigger earlier expiration
|
||||
|
||||
### Refreshing Sessions
|
||||
|
||||
**Automatic Refresh:** CSRF tokens and session IDs are automatically refreshed when authentication errors are detected. This handles most "session expired" errors transparently.
|
||||
|
||||
**Manual Re-authentication:** If your session cookies have fully expired (automatic refresh won't help), re-authenticate:
|
||||
|
||||
```bash
|
||||
notebooklm login
|
||||
```
|
||||
|
||||
### Multiple Accounts
|
||||
|
||||
**Profiles (recommended):** Use named profiles to manage multiple Google accounts under a single home directory:
|
||||
|
||||
```bash
|
||||
# Create and authenticate profiles
|
||||
notebooklm profile create work
|
||||
notebooklm -p work login
|
||||
notebooklm -p work list
|
||||
|
||||
notebooklm profile create personal
|
||||
notebooklm -p personal login
|
||||
notebooklm -p personal list
|
||||
|
||||
# Switch the active profile
|
||||
notebooklm profile switch work
|
||||
notebooklm list # Uses work profile
|
||||
|
||||
# List all profiles
|
||||
notebooklm profile list
|
||||
|
||||
# Use env var for session-wide override
|
||||
export NOTEBOOKLM_PROFILE=personal
|
||||
notebooklm list # Uses personal profile
|
||||
```
|
||||
|
||||
Each profile stores its own `storage_state.json`, `context.json`, and `browser_profile/` under `~/.notebooklm/profiles/<name>/`.
|
||||
|
||||
**Alternative: `NOTEBOOKLM_HOME`** still works for full directory-level isolation:
|
||||
|
||||
```bash
|
||||
export NOTEBOOKLM_HOME=~/.notebooklm-work
|
||||
notebooklm login
|
||||
```
|
||||
|
||||
**One-off override with `--storage`:**
|
||||
|
||||
```bash
|
||||
notebooklm --storage /path/to/storage_state.json list
|
||||
```
|
||||
|
||||
When `--storage <path>` is set, **two different context files** are used —
|
||||
they are NOT the same file:
|
||||
|
||||
- **Notebook / conversation context** lives at a *suffixed* file
|
||||
`<path>.context.json` (`storage_path.with_suffix(storage_path.suffix + ".context.json")`,
|
||||
see `paths.get_context_path`). Two `--storage` invocations against different
|
||||
files cannot see each other's selected notebook, and neither pollutes the
|
||||
default profile context.
|
||||
- **Account-routing metadata** (the `notebooklm.account` object — `authuser`
|
||||
index and optional `email`) lives in-band inside the selected
|
||||
`storage_state.json`. This keeps copied files and `NOTEBOOKLM_AUTH_JSON`
|
||||
secrets bound to the same Google account route as the original profile.
|
||||
|
||||
Run `notebooklm --storage <path> status --paths` to see exactly which
|
||||
context file is being used for notebook selection.
|
||||
|
||||
## CI/CD Configuration
|
||||
|
||||
### GitHub Actions (Recommended)
|
||||
|
||||
Use `NOTEBOOKLM_AUTH_JSON` for secure, file-free authentication:
|
||||
|
||||
```yaml
|
||||
jobs:
|
||||
notebook-task:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
- name: Setup Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
|
||||
- name: Install notebooklm-py
|
||||
run: pip install notebooklm-py
|
||||
|
||||
# Pre-flight: fail fast and loud on missing/expired auth.
|
||||
# `auth check --json` returns exit 0 even when status is "error"; --test makes the network
|
||||
# call needed to detect expired cookies, and the `jq -e` flag converts a non-"ok" status
|
||||
# into a non-zero exit code so the runner step actually fails.
|
||||
- name: Verify auth (fail-fast on expired cookies)
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
run: notebooklm auth check --test --json | jq -e '.status == "ok"'
|
||||
|
||||
- name: List notebooks
|
||||
env:
|
||||
NOTEBOOKLM_AUTH_JSON: ${{ secrets.NOTEBOOKLM_AUTH_JSON }}
|
||||
run: notebooklm list
|
||||
```
|
||||
|
||||
**Benefits:**
|
||||
- No file writes needed
|
||||
- Secret stays in memory only
|
||||
- Clean, simple workflow
|
||||
|
||||
### Obtaining the Secret Value
|
||||
|
||||
1. Run `notebooklm login` locally
|
||||
2. If the file was created by an older Playwright login and lacks `notebooklm.account`, run `notebooklm auth refresh` to repair single-account states or re-login with `notebooklm login --browser-cookies <browser> --account EMAIL` for multi-account states
|
||||
3. Copy the contents of `~/.notebooklm/profiles/default/storage_state.json` (the canonical write location; the legacy `~/.notebooklm/storage_state.json` is only read as a fallback)
|
||||
4. Add as a GitHub repository secret named `NOTEBOOKLM_AUTH_JSON` (see [installation.md#d-headless-server-or-ci](installation.md#d-headless-server-or-ci) for trailing-newline + ephemeral-runner refresh notes)
|
||||
|
||||
### Alternative: File-Based Auth
|
||||
|
||||
If you prefer file-based authentication:
|
||||
|
||||
```yaml
|
||||
- name: Setup NotebookLM auth
|
||||
run: |
|
||||
mkdir -p ~/.notebooklm/profiles/default
|
||||
echo "${{ secrets.NOTEBOOKLM_AUTH_JSON }}" > ~/.notebooklm/profiles/default/storage_state.json
|
||||
chmod 600 ~/.notebooklm/profiles/default/storage_state.json
|
||||
|
||||
- name: List notebooks
|
||||
run: notebooklm list
|
||||
```
|
||||
|
||||
For profile-specific CI auth:
|
||||
|
||||
```yaml
|
||||
- name: Setup work profile auth
|
||||
run: |
|
||||
mkdir -p ~/.notebooklm/profiles/work
|
||||
echo "${{ secrets.WORK_AUTH_JSON }}" > ~/.notebooklm/profiles/work/storage_state.json
|
||||
chmod 600 ~/.notebooklm/profiles/work/storage_state.json
|
||||
|
||||
- name: List notebooks (work)
|
||||
run: notebooklm -p work list
|
||||
```
|
||||
|
||||
### Session Expiration
|
||||
|
||||
CSRF tokens are automatically refreshed during API calls. However, the underlying session cookies still expire. For long-running CI pipelines:
|
||||
- Update the `NOTEBOOKLM_AUTH_JSON` secret every 1-2 weeks
|
||||
- Monitor for persistent auth failures (these indicate cookie expiration)
|
||||
|
||||
## Debugging
|
||||
|
||||
### Enable Verbose Output
|
||||
|
||||
Some commands support verbose output via Rich console:
|
||||
|
||||
```bash
|
||||
# Most errors are printed to stderr with details
|
||||
notebooklm list 2>&1 | cat
|
||||
```
|
||||
|
||||
### Enable RPC Debug Logging
|
||||
|
||||
```bash
|
||||
NOTEBOOKLM_DEBUG_RPC=1 notebooklm list
|
||||
```
|
||||
|
||||
### Logger namespace compatibility
|
||||
|
||||
Runtime transport and middleware logs still use the historical
|
||||
`notebooklm._core` logger key via `CORE_LOGGER_NAME`. That name is a
|
||||
compatibility logging contract for existing filters and tests; it does not
|
||||
mean the deleted `_core.py` module or a concrete `Session` owner still exists.
|
||||
|
||||
### Check Authentication
|
||||
|
||||
Verify your session is working:
|
||||
|
||||
```bash
|
||||
# Should list notebooks or show empty list
|
||||
notebooklm list
|
||||
|
||||
# If you see "Unauthorized" or redirect errors, re-login
|
||||
notebooklm login
|
||||
```
|
||||
|
||||
### Check Configuration Paths
|
||||
|
||||
```bash
|
||||
# See where files are being read from
|
||||
notebooklm status --paths
|
||||
```
|
||||
|
||||
### Network Issues
|
||||
|
||||
The CLI uses `httpx` for HTTP requests. Common issues:
|
||||
|
||||
- **Timeout**: Google's API can be slow; large operations may time out
|
||||
- **SSL errors**: Ensure your system certificates are up to date
|
||||
- **Proxy**: Set standard environment variables (`HTTP_PROXY`, `HTTPS_PROXY`) if needed
|
||||
|
||||
## Platform Notes
|
||||
|
||||
### macOS
|
||||
|
||||
Works out of the box. Chromium is downloaded automatically by Playwright.
|
||||
|
||||
### Linux
|
||||
|
||||
For Playwright system dependencies and the Chromium install on Debian/Ubuntu, see [docs/installation.md#platform-notes](installation.md#platform-notes) (and [troubleshooting.md#linux](troubleshooting.md#linux) if you hit `TypeError: onExit is not a function`).
|
||||
|
||||
### Windows
|
||||
|
||||
Works with PowerShell or CMD. Use backslashes for paths:
|
||||
|
||||
```powershell
|
||||
notebooklm --storage C:\Users\Name\.notebooklm\storage_state.json list
|
||||
```
|
||||
|
||||
Or set environment variable:
|
||||
|
||||
```powershell
|
||||
$env:NOTEBOOKLM_HOME = "C:\Users\Name\custom-notebooklm"
|
||||
notebooklm list
|
||||
```
|
||||
|
||||
### WSL
|
||||
|
||||
Browser login opens in the Windows host browser. The storage file is saved in the WSL filesystem.
|
||||
|
||||
### Headless Servers & Containers
|
||||
|
||||
**Playwright is only required for the `notebooklm login` command.** All other operations use standard HTTP requests via `httpx`.
|
||||
|
||||
For the install + auth-bootstrap recipe (run `notebooklm login` on a workstation, copy `storage_state.json` to the server, set `NOTEBOOKLM_AUTH_JSON`), see the canonical Persona D guide: [docs/installation.md#d-headless-server-or-ci](installation.md#d-headless-server-or-ci).
|
||||
@@ -0,0 +1,287 @@
|
||||
# Naming Conventions
|
||||
|
||||
**Status:** Active
|
||||
**Last Updated:** 2026-05-21
|
||||
|
||||
This document is the canonical reference for naming patterns that recur across
|
||||
the `notebooklm-py` codebase. It catalogues three families that an internal
|
||||
architecture audit (findings CC2 / CC3 / CC5) called out as inconsistent enough
|
||||
to need a written tiebreaker:
|
||||
|
||||
1. [Waiting / polling verbs](#1-waiting--polling-verbs-cc2) — `poll_X` vs
|
||||
`wait_for_X` vs `wait_until_X` vs `await_X` vs `_wait_for_X`.
|
||||
2. [RPC-callable Protocol names](#2-rpc-callable-protocol-names-cc3) —
|
||||
`NextCall` / `RpcCallback` / `RpcCaller`.
|
||||
3. [Metrics method verbs](#3-metrics-method-verbs-cc5) — `record_X` vs `emit_X`.
|
||||
|
||||
Examples below cite **symbol names only** (no file:line refs). Use
|
||||
`rg '<symbol>' src/notebooklm/` to locate the current home — line numbers drift
|
||||
faster than this file can keep up with.
|
||||
|
||||
---
|
||||
|
||||
## 1. Waiting / polling verbs (CC2)
|
||||
|
||||
Five distinct verbs are intentional. They are not interchangeable. Pick the
|
||||
shape that matches what the function actually does and the loop will document
|
||||
itself.
|
||||
|
||||
### `poll_X` — one-shot status read, no loop, no sleep
|
||||
|
||||
A `poll_X` function performs **a single** status / readiness check and returns
|
||||
immediately. It never sleeps and never iterates. Use this when the *loop* lives
|
||||
in the caller (or in a `wait_*` wrapper) and the function is just the
|
||||
per-iteration probe.
|
||||
|
||||
Examples:
|
||||
|
||||
- `ArtifactPollingService.poll_status` — single RPC list + scan for one task ID.
|
||||
- `ArtifactsAPI.poll_status` — public single-shot facade over the service.
|
||||
- `ResearchAPI.poll` — single status read for a research plan.
|
||||
- `artifact_poll` (CLI command) — one shot, then exit. Use the separate
|
||||
`artifact wait` command for the blocking / looping variant; `artifact poll`
|
||||
itself has no `--wait` flag.
|
||||
|
||||
> **Test name:** "if I call this twice in a row without a sleep, does that make
|
||||
> sense?" If yes → it's a `poll_X`.
|
||||
|
||||
### `wait_for_X` — bounded loop with a **timeout**
|
||||
|
||||
A `wait_for_X` function loops until either the awaited condition holds **or** a
|
||||
deadline expires. Timeouts are required (default or explicit); the function
|
||||
raises a typed `*TimeoutError` on expiry rather than returning a sentinel.
|
||||
|
||||
Examples:
|
||||
|
||||
- `ArtifactPollingService.wait_for_completion` — loops `poll_status` until the
|
||||
artifact is terminal or `timeout` elapses.
|
||||
- `ArtifactsAPI.wait_for_completion` — public facade over the service loop.
|
||||
- `ResearchAPI.wait_for_completion` — loops `poll` until research is terminal,
|
||||
pinning a discovered `task_id` between iterations.
|
||||
- `SourcePoller.wait_for_sources` (and `SourcesAPI.wait_for_sources`) — batch
|
||||
wait across N source IDs with a shared deadline.
|
||||
- `RetryMiddleware._wait_for_rate_limit` / `_wait_for_server_error` — private
|
||||
variant; see the underscore-prefix subsection below.
|
||||
|
||||
### `wait_until_X` — loop on a **predicate** (also bounded)
|
||||
|
||||
`wait_until_X` reads like English: "wait until `X` is true". Same loop+timeout
|
||||
contract as `wait_for_X`, but the verb signals that the awaited condition is a
|
||||
**state predicate** on a specific resource, not the *arrival* of a value.
|
||||
|
||||
Examples:
|
||||
|
||||
- `SourcePoller.wait_until_ready` / `SourcesAPI.wait_until_ready` — block until
|
||||
`source.is_ready`.
|
||||
- `SourcePoller.wait_until_registered` / `SourcesAPI.wait_until_registered` —
|
||||
block until a freshly-added source appears in the notebook listing.
|
||||
|
||||
> **`wait_for_X` vs `wait_until_X`:** both loop with a timeout. The difference
|
||||
> is naming ergonomics. Prefer `wait_until_X` when the awaited condition is a
|
||||
> boolean property of an existing resource (`is_ready`, `is_registered`).
|
||||
> Prefer `wait_for_X` when you're waiting on an external arrival or a *set* of
|
||||
> items (`wait_for_sources`, `wait_for_completion`). Neither form is "more
|
||||
> correct"; pick the one that reads naturally at the call site.
|
||||
|
||||
### `await_X` — coalesced single-flight join
|
||||
|
||||
`await_X` is reserved for **single-flight coalescing** primitives: many
|
||||
concurrent callers join one shared in-flight operation. The function name
|
||||
matches the user-facing verb ("await the refresh"), and the implementation
|
||||
guarantees deduplication (typically via `asyncio.shield` + a stored task).
|
||||
|
||||
Examples:
|
||||
|
||||
- `AuthRefreshCoordinator.await_refresh` — thundering-herd-safe token refresh;
|
||||
all 401-bouncing callers join one refresh task.
|
||||
|
||||
Do **not** use `await_X` for ordinary `async def` functions just because they
|
||||
get `await`-ed. The verb signals coalescing semantics, not async-ness.
|
||||
|
||||
### `_wait_for_X` — module-private backoff helper
|
||||
|
||||
The leading underscore + `wait_for_` shape is used inside middlewares to
|
||||
indicate **"this is the bounded backoff helper I extracted from one specific
|
||||
retry leg"**. It is not a public coordination primitive; it is a private
|
||||
implementation detail of a larger retry loop.
|
||||
|
||||
Examples:
|
||||
|
||||
- `RetryMiddleware._wait_for_rate_limit` — honors `Retry-After`, falls back to
|
||||
exponential backoff. Called from inside the rate-limit branch of the retry
|
||||
loop; never called externally.
|
||||
- `RetryMiddleware._wait_for_server_error` — same shape for the 5xx branch.
|
||||
|
||||
If you extract a backoff helper from a middleware, follow this pattern. If you
|
||||
extract a *public* waiting primitive, drop the underscore and use one of the
|
||||
four verbs above.
|
||||
|
||||
### Summary table
|
||||
|
||||
| Verb | Loop? | Timeout? | Predicate or arrival? | Shared single-flight? | Public? |
|
||||
|---|---|---|---|---|---|
|
||||
| `poll_X` | no (one-shot) | n/a | either | n/a | yes |
|
||||
| `wait_for_X` | yes | required | arrival of value(s) | no | yes |
|
||||
| `wait_until_X` | yes | required | state predicate | no | yes |
|
||||
| `await_X` | yes (joins one task) | inherits from task | n/a | yes | yes |
|
||||
| `_wait_for_X` | yes | required | arrival | no | no (module-private) |
|
||||
|
||||
---
|
||||
|
||||
## 2. RPC-callable Protocol names (CC3)
|
||||
|
||||
Most feature modules type their RPC dependency as the shared
|
||||
`RpcCaller` object Protocol from `_runtime/contracts.py`. Only middleware-chain
|
||||
callables and upload's keyword-injected registration callback keep local
|
||||
callable shapes. These names are NOT interchangeable — the divergence is
|
||||
structural, not stylistic. This section explains what each name signals so new
|
||||
code picks the right shape.
|
||||
|
||||
### The three names in use
|
||||
|
||||
| Name | Defined in | Protocol shape | Used by |
|
||||
|---|---|---|---|
|
||||
| `NextCall` | `_middleware/core.py` | **type alias**, not a class: `Callable[[RpcRequest], Awaitable[RpcResponse]]` | Every `Middleware.__call__` — the "call the next link" function passed into around-style middlewares |
|
||||
| `RpcCallback` | `_source/upload.py` | **Callable** Protocol: `async def __call__(method, params, ...)` | `SourceUploadPipeline.register_file_source` — RPC entrypoint passed as a **keyword argument** at call time |
|
||||
| `RpcCaller` | `_runtime/contracts.py` | **Object** Protocol: `async def rpc_call(method, params, ...)` (i.e. `obj.rpc_call(...)`) | The canonical shared capability Protocol for pure-RPC feature APIs and helper services (`NotesAPI`, `SourceLister`, `ShareManager`, etc.) |
|
||||
|
||||
### Why they diverge
|
||||
|
||||
Two axes do the actual work:
|
||||
|
||||
1. **Callable Protocol vs Object Protocol.** `NextCall` and `RpcCallback` are
|
||||
*callable* shapes — the conformer is itself directly invokable
|
||||
(`rpc(method, params)`). `RpcCaller` is an
|
||||
*object* shape — the conformer exposes an `.rpc_call(...)` method
|
||||
(`executor.rpc_call(method, params)`).
|
||||
These are NOT interchangeable to mypy: a callable Protocol matches a bare
|
||||
function or `__call__`, while `RpcCaller` requires the named method. An
|
||||
`RpcExecutor` instance satisfies `RpcCaller` because it defines `rpc_call`; the
|
||||
bound method `executor.rpc_call` satisfies `RpcCallback` because it is a
|
||||
callable Protocol.
|
||||
2. **Type alias vs Protocol class.** `NextCall` is a `Callable[...]` alias, not
|
||||
a class. It exists because the middleware chain is built from a list of
|
||||
wrapped callables (`functools.reduce`-style composition); a Protocol class
|
||||
would not buy anything over the alias and would make the middleware
|
||||
constructor signatures noisier.
|
||||
|
||||
`RpcCallback` exists separately from `RpcCaller` for one remaining reason:
|
||||
it is a **keyword-only callback** passed into `register_file_source`, and
|
||||
keeping it as a structural Protocol (instead of a bare `Callable[...]` alias)
|
||||
lets mypy flag keyword-name typos at the call site.
|
||||
|
||||
### Choosing a name in new code
|
||||
|
||||
- New pure-RPC feature API? Type the dependency as
|
||||
**`RpcCaller`** from `_runtime/contracts.py`. This is the shared capability
|
||||
Protocol; see [`docs/architecture.md`](./architecture.md) for the protocol
|
||||
catalogue. The concrete `RpcExecutor` and `NotebookLMClient` satisfy it
|
||||
structurally.
|
||||
- New middleware? Use **`NextCall`** from `_middleware/core.py` for the chain
|
||||
callable — do not invent a new alias.
|
||||
- New feature that takes the RPC entrypoint as a **keyword argument** at call
|
||||
time? Define a local Protocol named **`RpcCallback`** so the keyword-typo
|
||||
detection kicks in at every call site.
|
||||
|
||||
> **Why not collapse the last callback too?** `SourceUploadPipeline` accepts
|
||||
> `rpc_call=` as a keyword override inside `register_file_source(...)`; keeping
|
||||
> a callable Protocol there preserves mypy's keyword-name checking at the
|
||||
> override seam. Ordinary constructor-injected feature services should use
|
||||
> `RpcCaller`.
|
||||
|
||||
This convention is guarded by
|
||||
`tests/_guardrails/test_no_legacy_rpc_callable_aliases.py`: `RpcCall` and `ShareRpc`
|
||||
must stay deleted, and `RpcCallback` must stay local to `_source/upload.py`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Metrics method verbs (CC5)
|
||||
|
||||
`ClientMetrics` exposes two verb families. They have different threading and
|
||||
back-pressure contracts; the verb is the contract.
|
||||
|
||||
### `record_X` — sync, mutates counter state under a lock
|
||||
|
||||
`record_X` methods are **synchronous**, take a numeric measurement (typically
|
||||
seconds), and update an in-memory `ClientMetricsSnapshot` field under
|
||||
`_metrics_lock`. They never call user code, never schedule tasks, and never
|
||||
block on I/O. Callers do not need to be inside an event loop.
|
||||
|
||||
Examples (all on `ClientMetrics`):
|
||||
|
||||
- `record_rpc_queue_wait` — time waiting for the RPC semaphore.
|
||||
- `record_upload_queue_wait` — time waiting for the upload semaphore.
|
||||
- `record_lock_wait` — time waiting on `_reqid_lock` (or a similar internal
|
||||
lock).
|
||||
|
||||
Shared backend: `ClientMetrics._record_wait` (private; the three public
|
||||
methods are typed wrappers around it).
|
||||
|
||||
> Upload queue waits are recorded through the `ClientMetrics` instance injected
|
||||
> into the upload pipeline. The verb stays `record_*` because the contract is
|
||||
> still sync + lock-protected counter mutation.
|
||||
|
||||
### `emit_X` — async, fires the user-supplied callback
|
||||
|
||||
`emit_X` methods are **`async def`** and `await` the user-configured telemetry
|
||||
callback (`on_rpc_event=...`). They are the back-pressure seam: the awaited
|
||||
callback can hold up the calling RPC if it does I/O, and that is intentional
|
||||
(rate-limit feedback flows backwards into the producer).
|
||||
|
||||
Examples:
|
||||
|
||||
- `ClientMetrics.emit_rpc_event` — awaits the `on_rpc_event` callback with a
|
||||
`RpcTelemetryEvent` payload; swallows + logs callback exceptions so a
|
||||
misbehaving callback can't corrupt the RPC path.
|
||||
|
||||
Exceptions inside the callback are caught and logged (observability must not
|
||||
alter behavior), but the `await` itself is load-bearing — *don't* fire-and-forget
|
||||
the callback with `asyncio.create_task(...)`, because that would defeat the
|
||||
back-pressure contract.
|
||||
|
||||
### Choosing a verb in new code
|
||||
|
||||
- The new method updates a counter / gauge / histogram bucket synchronously?
|
||||
Use **`record_X`** and document the unit (seconds, bytes, count).
|
||||
- The new method dispatches an event to user code (callback, queue, log
|
||||
sink) and the producer should `await` it? Use **`emit_X`** and make it
|
||||
`async def`.
|
||||
- If both apply (record *and* emit), do them as two calls: `record_*` first
|
||||
(cheap, lock-protected), then `await emit_*` (potentially slow, can raise —
|
||||
though `ClientMetrics.emit_rpc_event` swallows). Keep the verbs separate so
|
||||
the contract at each call site stays one line of code.
|
||||
|
||||
---
|
||||
|
||||
## 4. Historical-work vocabulary (glossary)
|
||||
|
||||
Comments, ADRs, `docs/refactor-history.md`, and `CLAUDE.md` use a handful of
|
||||
recurring labels to reference past refactoring work. They are accurate but
|
||||
opaque without a key. When you write a *new* comment, prefer a one-line
|
||||
behavioral rationale over one of these labels — but when you read an existing
|
||||
one, this is what it means:
|
||||
|
||||
| Token | Meaning |
|
||||
|-------|---------|
|
||||
| **`#NNNN`** | A GitHub pull request or issue number in `teng-lin/notebooklm-py`, e.g. <https://github.com/teng-lin/notebooklm-py/pull/1205>. Bare numbers are not clickable in most renderers; resolve them on GitHub. |
|
||||
| **ADR-NNNN** | An Architecture Decision Record under [`docs/adr/`](./adr/) (zero-padded, e.g. ADR-0014 -> `docs/adr/0014-*.md`). The authoritative source for a design decision and its trade-offs. |
|
||||
| **Wave N** | A sequenced step of a larger multi-PR migration (e.g. the session-decoupling / host-protocol-removal effort). Waves are *planning* units, not releases; the durable record of what a wave did is the ADR it closed and the merged PRs. |
|
||||
| **Phase N** | A coarser planning grouping than a Wave, used by some older migrations. |
|
||||
| **Tier N** | Yet another planning grouping (e.g. "tier-12-13" migration); interchangeable with Phase in intent. |
|
||||
| **`Pn.Tn`** (e.g. `P3.T0`) | "Phase *n*, Task *n*" task codes from a numbered refactor plan; used in `CLAUDE.md` to date module renames. |
|
||||
| **`.sisyphus/plans/…`** | Internal planning notes that are **not checked into the repo**. References to them are unfollowable by outside readers; treat them as provenance only. New code should cite the public ADR or doc instead. |
|
||||
|
||||
When historical labels are the *only* explanation for why code exists, that is a
|
||||
smell — promote the reasoning into a plain-language comment or an ADR so the
|
||||
rationale survives without the label.
|
||||
|
||||
## Related documents
|
||||
|
||||
- [`docs/architecture.md`](./architecture.md) — the current runtime graph,
|
||||
capability Protocols, and the `RpcCaller` catalogue entry.
|
||||
- [`docs/development.md`](./development.md) — contributor on-ramp; this
|
||||
conventions doc is linked from its "Key Design Decisions" section.
|
||||
- The architecture audit that motivated this catalogue lives in internal
|
||||
planning notes that are not checked into the repo. Future codebase audits
|
||||
with naming-convention findings should
|
||||
extend this document rather than spawn parallel tiebreaker files.
|
||||
@@ -0,0 +1,145 @@
|
||||
# Deprecations
|
||||
|
||||
This page is the **single source of truth** for currently-deprecated APIs in
|
||||
`notebooklm-py`. Each row lists what is deprecated, the recommended
|
||||
replacement, when the deprecation was introduced, when it is scheduled for
|
||||
removal, and any cross-references.
|
||||
|
||||
`docs/stability.md` links here rather than duplicating the table; if you need
|
||||
the broader stability policy (semver promise, supported Python versions, the
|
||||
0.x pre-1.0 semantics), start there.
|
||||
|
||||
> **Upgrading to v0.8.0?** The breaking error-and-return contract changes
|
||||
> **shipped in v0.8.0**. See the consolidated
|
||||
> [Upgrading to v0.8.0](upgrading-to-0.8.0.md) guide for the full set and the
|
||||
> exact before→after migration for each. The `NOTEBOOKLM_FUTURE_ERRORS` preview
|
||||
> flag that staged these changes in v0.7.0 has been **removed** (it is now a
|
||||
> no-op).
|
||||
|
||||
## Scheduled for removal
|
||||
|
||||
| Deprecated | Replacement | Since | Removal | Notes |
|
||||
|------------|-------------|-------|---------|-------|
|
||||
| Awaiting `NotebookLMClient.from_storage(...)` | `async with NotebookLMClient.from_storage(...) as client:` | v0.5.0 | v1.0 | The `__await__` form still works. Warning emitted via `src/notebooklm/_deprecation.py::warn_deprecated`; suppress with `NOTEBOOKLM_QUIET_DEPRECATIONS=1` ([#1369](https://github.com/teng-lin/notebooklm-py/issues/1369)) |
|
||||
| MCP `research_status(task_id=…)` / `research_import(task_id=…)` / `research_cancel(run_id=…)` | The same value under `poll_task_id=…` on all three | v0.8.0 | v0.9.0 | The three tools each accept the id that `research_start` / `research_status` surface as `poll_task_id` — renamed so the value copies verbatim between tools. The old `task_id` / `run_id` param names still work as aliases but emit a `DeprecationWarning` (via `warn_deprecated`) and add a `deprecation` note to the tool result; passing both names with different values is a validation error. ([#1789](https://github.com/teng-lin/notebooklm-py/issues/1789)) |
|
||||
|
||||
> The v0.8.0 error-contract runways (`get()`-returns-`None`, the
|
||||
> `wait_for_completion(interval=...)` alias, the dict-subscript bridge,
|
||||
> `NotebooksAPI.share()`, and the ambiguous `research.poll(task_id=None)` guard)
|
||||
> all completed their removal cycle in **v0.8.0** — see
|
||||
> [Removed in v0.8.0](#removed-in-v080) below.
|
||||
|
||||
## Removed in v0.8.0
|
||||
|
||||
These error-and-return contract changes completed their v0.7.0 deprecation /
|
||||
preview cycle and are now the **default** behavior. The full before→after
|
||||
migration for each is in
|
||||
[`docs/upgrading-to-0.8.0.md`](upgrading-to-0.8.0.md).
|
||||
|
||||
| Removed | Replacement | Deprecated since | Removed in | Notes |
|
||||
|---------|-------------|------------------|------------|-------|
|
||||
| `sources.get()` / `artifacts.get()` / `notes.get()` / `mind_maps.get()` returning `None` on a miss | `get_or_none()` (warning-free `None`-on-miss), or `try/except SourceNotFoundError` / `ArtifactNotFoundError` / `NoteNotFoundError` / `MindMapNotFoundError` | v0.7.0 | v0.8.0 | A miss now **raises** the matching `*NotFoundError`, unifying the not-found contract with `notebooks.get()`; return annotations narrow from `X \| None` to `X`. The v0.7.0 `DeprecationWarning` (and the `warn_get_returns_none` helper) are gone. [#1247](https://github.com/teng-lin/notebooklm-py/issues/1247) |
|
||||
| Dict-subscript access (`result["status"]`) on `research.poll` / `research.start` / `research.wait_for_completion`, `artifacts.generate_mind_map`, and `sources.get_guide` returns | Attribute access (`result.status`, `result.sources`, `guide.summary`, …) | v0.7.0 | v0.8.0 | The typed returns (`ResearchTask` / `ResearchStart` / `MindMapResult` / `SourceGuide` / `ResearchSource`) are now pure attribute-only frozen dataclasses: `result["key"]` raises `TypeError`; `result.get(...)` / `.keys()` / `.items()` / `.values()` raise `AttributeError`; `"k" in result` / `iter(result)` / `len(result)` raise `TypeError`. Only attribute access and `to_public_dict()` survive. `ResearchStatus` stays a `str`-enum, so `status == "completed"` keeps working. The `MappingCompatMixin` bridge is removed. [#1251](https://github.com/teng-lin/notebooklm-py/issues/1251) |
|
||||
| `ResearchAPI.wait_for_completion(interval=...)` | `initial_interval=...` — same cadence, matching `SourcesAPI.wait_until_ready` / `ArtifactsAPI.wait_for_completion` | v0.7.0 | v0.8.0 | The deprecated `interval=` keyword alias is gone; passing it now raises the standard `TypeError` for an unexpected keyword. The `deprecated_kwarg` helper that powered the alias is removed. [#1254](https://github.com/teng-lin/notebooklm-py/issues/1254) |
|
||||
| `sources.refresh()` / `chat.delete_conversation()` returning `True` | (no replacement — discard the value) | n/a (clean break) | v0.8.0 | Both now return `None`; their annotations change from `-> bool` to `-> None`. The `True` carried no information (any failure raised first). `chat.clear_cache(...)` is unchanged and stays `-> bool`. [#1290](https://github.com/teng-lin/notebooklm-py/issues/1290) |
|
||||
| Synchronous generation-kickoff refusal swallowed into `GenerationStatus(status="failed")` / returned `None` | Catch the re-raised `RateLimitError` / `RPCError` / `DecodingError` / `ArtifactFeatureUnavailableError` | n/a (clean break) | v0.8.0 | `generate_*` / `revise_slide` / `_parse_generation_result` / `research.start` now **raise** on a "couldn't-start" refusal instead of returning a soft-failed status. `research.start`'s return narrows from `ResearchStart \| None` to `ResearchStart`; `with_rate_limit_retry` retries only on a raised `RateLimitError`. [#1342](https://github.com/teng-lin/notebooklm-py/issues/1342) |
|
||||
| Derived-read / lister drift collapsing malformed payloads to empty / `None` | Catch the raised `DecodingError` (distinct from a genuine miss) | n/a (clean break) | v0.8.0 | `sources.check_freshness()`, the note lister, and the artifact raw lister now raise `DecodingError` on a structurally-unrecognized payload. Legitimate empty / stale shapes are unchanged. [#1344](https://github.com/teng-lin/notebooklm-py/issues/1344) |
|
||||
| `notes.update()` / `sources.rename(return_object=False)` / `artifacts.rename(return_object=False)` silently succeeding on a missing target | Catch the raised `*NotFoundError` | n/a (clean break) | v0.8.0 | These now run an existence preflight and raise `NoteNotFoundError` / `SourceNotFoundError` / `ArtifactNotFoundError` on a miss. `return_object=False` still returns `None` on success. [#1362](https://github.com/teng-lin/notebooklm-py/issues/1362) |
|
||||
| `NotebooksAPI.share()` | `client.sharing.set_public()` + `client.notebooks.get_share_url()` | v0.5.0 | v0.8.0 | The deprecated no-behavior-change wrapper is removed. [#1363](https://github.com/teng-lin/notebooklm-py/issues/1363) |
|
||||
| `ResearchAPI.poll(task_id=None)` / `wait_for_completion(task_id=None)` silently guessing among multiple in-flight tasks | Pass the explicit `task_id` from `research.start` | v0.6.0 | v0.8.0 | With two or more tasks in flight these now raise the new `AmbiguousResearchTaskError` instead of warning and returning the latest task; with a single in-flight task they resolve it silently. [#1363](https://github.com/teng-lin/notebooklm-py/issues/1363) |
|
||||
| `NOTEBOOKLM_FUTURE_ERRORS` opt-in preview flag | (no replacement — the previewed behavior is now the default) | v0.7.0 | v0.8.0 | The forward-compat preview gate is removed; setting it is a no-op. The dict-subscript / get-returns-`None` / kwarg-alias deprecation helpers it gated are deleted with it. [#1365](https://github.com/teng-lin/notebooklm-py/issues/1365) |
|
||||
| `SettingsAPI.get_account_tier()` + the `AccountTier` type (`notebooklm.AccountTier` / `notebooklm.types.AccountTier`) | `client.settings.get_account_limits()` — `AccountLimits.tier` for the subscription tier (since v0.9.0), plus `.notebook_limit` / `.source_limit` for quotas | n/a (clean break) | v0.8.0 | The tier came from `GET_USER_TIER` (live method `FetchRecommendations`, a **promotions** endpoint), a promotion-eligibility signal that could **not** distinguish free from paid — both free and Pro accounts reported `NOTEBOOKLM_TIER_PRO_CONSUMER_USER`. The authoritative quota signal is `AccountLimits`. **Update (v0.9.0):** a *correct* tier signal is now back as `AccountLimits.tier` — an opaque enum read from the authoritative `GET_USER_SETTINGS` limits block (index 4), not the promotions RPC — and the MCP/REST `server_info(include_account=True)` account block exposes a `tier` key again (the removed `plan_name` string does **not** return). |
|
||||
|
||||
> **`wait_timeout` was deliberately kept.** The `wait_timeout` keyword on the
|
||||
> `SourcesAPI.add_*` family (`add_url` / `add_text` / `add_file` / `add_drive`)
|
||||
> was **not** renamed to `timeout`: on those methods `timeout` would be ambiguous
|
||||
> with a per-request HTTP timeout, while `wait_timeout` reads as "how long to wait
|
||||
> for readiness after adding". `SourcesAPI.add_file(mime_type=...)` and
|
||||
> `notebooklm source add --mime-type` are likewise **not** deprecated —
|
||||
> `mime_type` sets the resumable-upload content-type header.
|
||||
|
||||
> **`notebooklm.rpc` public surface tightened — not a removal (v0.8.0,
|
||||
> [#1589](https://github.com/teng-lin/notebooklm-py/issues/1589)).**
|
||||
> `notebooklm.rpc.__all__` now advertises only the two documented power-user
|
||||
> imports, `RPCMethod` and `resolve_rpc_id`. The ~47 other names it used to list
|
||||
> — the batchexecute wire helpers (`encode_rpc_request`, `decode_response`,
|
||||
> `extract_rpc_result`, `safe_index`, …), the endpoint URL constants/helpers, and
|
||||
> the enum / exception **re-exports** — are **not removed**: they remain
|
||||
> importable as `notebooklm.rpc.<name>` for back-compat. They were never part of
|
||||
> the supported public API (`docs/stability.md` has always marked
|
||||
> `notebooklm.rpc.*` internal); this change only stops the compat gate from
|
||||
> advertising them. New code should import the canonical public name where one
|
||||
> exists: most enums as `notebooklm.<X>` / `notebooklm.types.<X>`, but
|
||||
> `ArtifactStatus` and `artifact_status_to_str` only as `notebooklm.types.<X>`;
|
||||
> the exceptions as `notebooklm.<X>` / `notebooklm.exceptions.<X>`. The wire
|
||||
> helpers, the endpoint URL constants/helpers, `safe_index`, `ArtifactTypeCode`,
|
||||
> and `RPCErrorCode` are internal with **no** blessed public alias and stay
|
||||
> importable only as `notebooklm.rpc.<name>`. For raw-RPC power use, import
|
||||
> `from notebooklm.rpc import RPCMethod, resolve_rpc_id`.
|
||||
|
||||
> **`notebooklm.auth` public surface tightened — not a removal (v0.8.0,
|
||||
> [#1592](https://github.com/teng-lin/notebooklm-py/issues/1592)).**
|
||||
> `auth.__all__` no longer advertises 23 internal re-exports that only first-party
|
||||
> `src`/tests imported (cookie-snapshot/storage helpers, the WIZ-extraction helpers,
|
||||
> `authuser_query`/`format_authuser_value`, `load_httpx_cookies`/`normalize_cookie_map`,
|
||||
> `ALLOWED_COOKIE_DOMAINS`/`MINIMUM_REQUIRED_COOKIES`, the keepalive/refresh env +
|
||||
> URL constants, `load_auth_from_storage`, `fetch_tokens`, `recover_psidts_in_memory`).
|
||||
> These were migration leftovers from the `_auth/*` extraction (ADR-0003 → ADR-0014).
|
||||
> They are **not removed**: each remains importable as `notebooklm.auth.<name>` for
|
||||
> back-compat — first-party code now imports them from their `notebooklm._auth.<sub>`
|
||||
> home. `notebooklm.auth.*` has always been internal (`docs/stability.md`) except the
|
||||
> documented imports (`AuthTokens`, `convert_rookiepy_cookies_to_storage_state`, the
|
||||
> cookie-domain constants) and the cohesive operations (`enumerate_accounts`,
|
||||
> `fetch_tokens_with_domains`, `fetch_tokens_passive`, …), which are unchanged. A
|
||||
> deeper service-interface refactor of the remaining cli/_app-forced names was
|
||||
> evaluated and deferred (limited encapsulation payoff while names stay importable).
|
||||
|
||||
|
||||
## Removed in v0.7.0
|
||||
|
||||
| Removed | Replacement | Deprecated since | Removed in | Notes |
|
||||
|---------|-------------|------------------|------------|-------|
|
||||
| `NOTEBOOKLM_STRICT_DECODE=0` soft-mode opt-out | Unset the variable (strict is the only mode) | v0.5.0 | v0.7.0 | The env var is now ignored; `safe_index` always raises `UnknownRPCMethodError` on shape drift. Rationale in `docs/stability.md` "Strict decode" + ADR-0011 |
|
||||
| Positional `wait` / `wait_timeout` on `SourcesAPI.add_url`, `SourcesAPI.add_text`, `SourcesAPI.add_file`, `SourcesAPI.add_drive` | Pass `wait=...` and `wait_timeout=...` as keywords | v0.5.0 | v0.7.0 | `wait` / `wait_timeout` are now keyword-only; positional calls raise `TypeError`. CLI already used keyword arguments |
|
||||
| `ArtifactsAPI.wait_for_completion(poll_interval=...)` | `initial_interval=...` — same cadence, clearer name | v0.5.0 | v0.7.0 | The `poll_interval` keyword was removed; passing it raises `TypeError` |
|
||||
| `NotesAPI.create_from_chat(...)` | `ChatAPI.save_answer_as_note(...)` | v0.5.0 | v0.7.0 | Pure deprecated forwarder, now removed (two MINOR cycles of warnings served). `ChatAPI.save_answer_as_note(...)` is the canonical citation-rich saved-from-chat method and data owner (ADR-0013); call it directly. |
|
||||
|
||||
## Removed in v0.6.0
|
||||
|
||||
| Removed | Replacement | Deprecated since | Removed in | Notes |
|
||||
|---------|-------------|------------------|------------|-------|
|
||||
| `NotebookLMClient.rpc_call(source_path=...)` | Omit the argument; the canonical `"/"` default is applied unconditionally | v0.5.0 | v0.6.0 | Public escape-hatch wrapper kept; only the kwarg was cut. No public replacement — callers that need a non-`"/"` source path should add a typed sub-client method (open an issue) rather than reaching across the wrapper. |
|
||||
| `NotebookLMClient.rpc_call(_is_retry=...)` | Omit the argument | v0.5.0 | v0.6.0 | Internal-only retry flag; never part of the supported public surface. |
|
||||
| `NotebookLMClient.rpc_call(operation_variant=...)` | Omit the argument | v0.5.0 | v0.6.0 | Internal-only routing key for the mutating-RPC idempotency registry. |
|
||||
|
||||
## How deprecations work in this project
|
||||
|
||||
* Every deprecated surface emits a `DeprecationWarning` from the call site
|
||||
the user wrote, so the warning's `filename`/`lineno` point at user code
|
||||
rather than at the library internals.
|
||||
* Default-shape calls remain silent. A deprecation only fires when the
|
||||
caller actually passes the deprecated argument or surface.
|
||||
* `NOTEBOOKLM_QUIET_DEPRECATIONS=1` suppresses **every** deprecation warning
|
||||
this project emits — the one-off warnings routed through
|
||||
`src/notebooklm/_deprecation.py::warn_deprecated` (e.g. awaiting
|
||||
`from_storage(...)`). All mechanics live in `_deprecation.py`; ADR-0018 forbids
|
||||
inline `warnings.warn(..., DeprecationWarning)` elsewhere and a lint
|
||||
(`tests/_guardrails/test_no_inline_deprecation_warnings.py`) enforces it. See
|
||||
`docs/configuration.md`.
|
||||
* Not every inline `warnings.warn(...)` is a deprecation. The
|
||||
`save_cookies_to_storage(original_snapshot=None)` legacy full-merge path is a
|
||||
*permanent* public-API back-compat shim (see
|
||||
`docs/auth-cookie-lifecycle.md` §3.4.1), not a scheduled removal, so it emits
|
||||
a **`RuntimeWarning`** safety advisory about the stale-overwrite-fresh race —
|
||||
outside ADR-0018's scope and intentionally **not** silenced by
|
||||
`NOTEBOOKLM_QUIET_DEPRECATIONS`.
|
||||
* `NOTEBOOKLM_FUTURE_ERRORS` was the v0.7.0 forward-compat preview gate for the
|
||||
v0.8.0 error contract; it was **removed in v0.8.0** now that every break it
|
||||
staged is the default, and setting it is a no-op.
|
||||
* See `docs/stability.md` "Deprecation Policy" for the broader timeline
|
||||
contract (one MINOR cycle of warnings before removal during 0.x).
|
||||
|
||||
## Removed in past versions
|
||||
|
||||
For deprecations that have already completed their removal cycle, see
|
||||
`docs/stability.md` "Removed in v0.5.0".
|
||||
+1106
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,588 @@
|
||||
# Installation
|
||||
|
||||
**Last Updated:** 2026-07-11
|
||||
|
||||
This is the canonical installation guide for `notebooklm-py`. The README has a quickstart; everything else lives here.
|
||||
|
||||
**Contents**
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Quick install (TL;DR by persona)](#quick-install-tldr-by-persona)
|
||||
- [Choose your install path](#choose-your-install-path)
|
||||
- [A. AI Agent (primary persona)](#a-ai-agent-primary-persona)
|
||||
- [B. End user](#b-end-user)
|
||||
- [C. Library user](#c-library-user)
|
||||
- [D. Headless server or CI](#d-headless-server-or-ci)
|
||||
- [E. Contributor](#e-contributor)
|
||||
- [F. Power user](#f-power-user)
|
||||
- [Optional extras matrix](#optional-extras-matrix)
|
||||
- [Post-install steps](#post-install-steps)
|
||||
- [Verifying your install](#verifying-your-install)
|
||||
- [Platform notes](#platform-notes)
|
||||
- [Upgrading and uninstalling](#upgrading-and-uninstalling)
|
||||
- [Common gotchas (appendix)](#common-gotchas-appendix)
|
||||
- [All vs All-Extras](#all-vs-all-extras)
|
||||
- [`uv pip install` vs `uv sync`](#uv-pip-install-vs-uv-sync)
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Python 3.10 or later.** Tested and classified for 3.10, 3.11, 3.12, 3.13, 3.14. The CLI hard-fails with a clear error on older versions (see `_version_check.py`).
|
||||
- **Operating systems.** macOS (primary development platform), Linux (Debian/Ubuntu, Fedora), Windows 10/11, WSL.
|
||||
- **`uv` (optional but recommended for contributors).** Install with `curl -LsSf https://astral.sh/uv/install.sh | sh` or `brew install uv` / `winget install astral-sh.uv`. End users can use plain `pip` or `pipx`.
|
||||
- **Disk and bandwidth.** Base install is small (~10 MB). The first `notebooklm login` downloads Chromium (~170 MB; 30–90 s; **no progress bar** — be patient).
|
||||
|
||||
---
|
||||
|
||||
## Quick install (TL;DR by persona)
|
||||
|
||||
> **Installing the CLI on macOS / Linux — use an isolated installer.** Plain
|
||||
> `pip install` into the *system* interpreter fails on modern macOS (Homebrew
|
||||
> Python) and Debian/Ubuntu with `error: externally-managed-environment`
|
||||
> ([PEP 668](https://peps.python.org/pep-0668/)). For CLI/app use, prefer
|
||||
> **`uv tool install`** or **`pipx install`** — they put `notebooklm` (and
|
||||
> `notebooklm-mcp` / `notebooklm-server`) on your PATH in a dedicated environment
|
||||
> without touching system Python. Plain `pip` still works **inside a virtualenv**
|
||||
> and on **Windows** (python.org's Python is not externally-managed). Library
|
||||
> users install into their own project's venv (`uv add` / `pip install`), so
|
||||
> PEP 668 never applies.
|
||||
|
||||
| Persona | Install command |
|
||||
|---|---|
|
||||
| **A — AI Agent** | `pip install "notebooklm-py[browser]"` in the user's active env (fall back to `uv tool install` / `pipx install` on an *externally-managed-environment* error) |
|
||||
| **B — End user** | `uv tool install "notebooklm-py[browser]"` or `pipx install "notebooklm-py[browser]"` (isolated; avoids the PEP 668 error) |
|
||||
| **C — Library user** | `uv add notebooklm-py` (or `pip install notebooklm-py` inside your project venv) |
|
||||
| **D — Headless server / CI** | `pip install notebooklm-py` inside a venv/container; ship a `storage_state.json` (no Playwright) |
|
||||
| **E — Contributor** | `uv sync --frozen --extra browser --extra dev --extra markdown && uv run playwright install chromium && uv run pre-commit install` |
|
||||
| **F — Power user** | `uv tool install --python 3.12 "notebooklm-py[browser,cookies,markdown]"` (the `cookies` extra needs Python ≤ 3.12; `--python 3.12` makes uv provision a matching interpreter even if your default is 3.13+) |
|
||||
|
||||
---
|
||||
|
||||
## Choose your install path
|
||||
|
||||
### A. AI Agent (primary persona)
|
||||
|
||||
For Claude Code, Codex, and similar agent harnesses.
|
||||
|
||||
The project ships `notebooklm skill install`, [SKILL.md](../SKILL.md), and [AGENTS.md](../AGENTS.md). Agents run install on the user's behalf in the user's existing environment — no new venv. They typically can't *interact* with a browser, but most agent harnesses (Claude Code, Codex) *can* shell out to Playwright when a user is present, and `[cookies]` is a preferred optimization for reusing the user's already-logged-in browser cookies.
|
||||
|
||||
> **Note on agent harness coverage:** `notebooklm skill install` empirically writes to `~/.claude/skills/notebooklm/SKILL.md` and `~/.agents/skills/notebooklm/SKILL.md`. Cursor and other harnesses with bespoke skill formats are not auto-targeted; they fall back to `pip install` + manual skill registration.
|
||||
|
||||
**Recommended install (Python-version-aware; surfaces real errors instead of swallowing them):**
|
||||
|
||||
<!-- not mirrored: end-user install path (Persona A); CONTRIBUTING.md tracks the in-repo `uv sync` flow only -->
|
||||
```bash
|
||||
pip install "notebooklm-py[browser]" # mandatory; errors must propagate
|
||||
|
||||
# [cookies] (rookiepy) is optional and known to FAIL TO BUILD on Python 3.13+.
|
||||
# Skip it deliberately on 3.13+ rather than swallowing the error — that lets
|
||||
# *real* install failures (typos, network, PyPI outages) surface for the agent.
|
||||
if python -c "import sys; sys.exit(0 if sys.version_info < (3, 13) else 1)"; then
|
||||
pip install "notebooklm-py[cookies]" # errors propagate
|
||||
else
|
||||
echo "Skipping [cookies] on Python 3.13+ (rookiepy unavailable). Use 'notebooklm login' interactively."
|
||||
fi
|
||||
```
|
||||
|
||||
> If `pip install` errors with `externally-managed-environment` (modern macOS / Debian system Python, [PEP 668](https://peps.python.org/pep-0668/)), retry with `uv tool install "notebooklm-py[browser]"` or `pipx install "notebooklm-py[browser]"` — isolated installs that don't touch system Python. Inside an active virtualenv, `pip` works as-is.
|
||||
|
||||
**Why two separate calls (not `[browser,cookies]`):** the combined form is atomic — if `rookiepy` fails to compile, the whole install fails and the user gets **nothing**. Splitting means `[browser]` always succeeds; `[cookies]` is recoverable.
|
||||
|
||||
**Skill install (separate from the Python package):**
|
||||
|
||||
<!-- not mirrored: agent skill registration; not part of the contributor install flow -->
|
||||
```bash
|
||||
notebooklm skill install # writes to ~/.claude/skills/, ~/.agents/skills/
|
||||
# OR (alternative ecosystem):
|
||||
npx skills add teng-lin/notebooklm-py
|
||||
```
|
||||
|
||||
If the agent is reading `SKILL.md` from inside an already-installed location (e.g. `~/.claude/skills/notebooklm/SKILL.md`), the skill is already present — you only need the Python package install + auth.
|
||||
|
||||
**Authentication — `notebooklm login` is the primary path:**
|
||||
|
||||
<!-- not mirrored: end-user auth setup; contributors usually source storage_state from a personal account -->
|
||||
```bash
|
||||
notebooklm login # primary: opens browser, user signs in to Google once
|
||||
```
|
||||
|
||||
After login, `storage_state.json` persists at `~/.notebooklm/profiles/default/storage_state.json` and is reused on every subsequent run. **Verify with `notebooklm auth check --test --json`** (require `"status": "ok"` AND `"checks.token_fetch": true` — bare `auth check --json` only proves the file parses, not that the cookies still authenticate against Google).
|
||||
|
||||
**Headless / sandboxed agent contexts** (no display, can't open a browser): use the cookie-extraction path instead, requires the `[cookies]` extra installed in step 2:
|
||||
|
||||
<!-- not mirrored: headless-agent auth path; out of scope for the contributor README -->
|
||||
```bash
|
||||
notebooklm login --browser-cookies auto # rookiepy autodetects an installed browser
|
||||
```
|
||||
|
||||
If the agent is in a no-display sandbox AND `[cookies]` isn't installed (Python 3.13+ skipped it), ask the user to run `notebooklm login` on a workstation and copy the resulting `~/.notebooklm/profiles/default/storage_state.json` to the agent's environment (or set `NOTEBOOKLM_AUTH_JSON`).
|
||||
|
||||
#### Sandboxed agents (Claude Cowork)
|
||||
|
||||
**Claude Cowork** (Anthropic's sandboxed desktop agent for non-developers) and similar no-display sandboxes are a special case of the headless path above: there is no browser for `notebooklm login`, and the sandbox resets between sessions. Two adjustments make everything except `login` work:
|
||||
|
||||
- **Bootstrap each session** with the base install — `[browser]`/Playwright is *not* needed here, only for `login` (which you run elsewhere, once):
|
||||
<!-- not mirrored: Cowork per-session bootstrap; not part of the contributor flow -->
|
||||
```bash
|
||||
pip install notebooklm-py # queries/generation/download need no extras
|
||||
```
|
||||
- **Reuse a host-generated `storage_state.json`.** Run `notebooklm login` once on a machine with a display, then bring the file into a sandbox-accessible folder. Point at it with the root `--storage` flag or `NOTEBOOKLM_AUTH_JSON` — the same mechanism as [Persona D](#d-headless-server-or-ci):
|
||||
<!-- not mirrored: Cowork auth reuse; not part of the contributor flow -->
|
||||
```bash
|
||||
notebooklm --storage /path/to/storage_state.json list # per-invocation flag
|
||||
# OR inline via env var (no file needed — e.g. from a Cowork-stored secret):
|
||||
export NOTEBOOKLM_AUTH_JSON="$(cat /path/to/storage_state.json)"
|
||||
notebooklm list
|
||||
```
|
||||
|
||||
> ⚠️ **Security:** `storage_state.json` and `NOTEBOOKLM_AUTH_JSON` are bearer credentials for your Google account — store the file `0600` or in the sandbox's secret store, never commit or log them, and `unset NOTEBOOKLM_AUTH_JSON` after use.
|
||||
|
||||
Pass an explicit `-n/--notebook <id>` on notebook-scoped commands — the selected-notebook context does not survive a session reset. If Cowork reads `~/.claude/skills/`, `notebooklm skill install` registers the skill there; otherwise build the uploadable archive on the host with `notebooklm skill package` (writes `notebooklm-skill.zip`; see [cli-reference.md](cli-reference.md#skill-commands-notebooklm-skill-cmd)) and add it via **Claude Settings → Capabilities**.
|
||||
|
||||
**Verification (machine-parseable):**
|
||||
|
||||
<!-- not mirrored: agent-targeted verification; CONTRIBUTING.md uses the test+lint suite as its smoke check -->
|
||||
```bash
|
||||
notebooklm --version # text version
|
||||
notebooklm auth check --json # JSON: {"status": "ok", "checks": {...}}
|
||||
notebooklm auth check --test --json # same + network token-fetch validation
|
||||
notebooklm list --json # JSON list (may be empty for new accounts)
|
||||
```
|
||||
|
||||
> **Important:** `notebooklm status` is *context state* (selected notebook), **NOT auth**. Do not grep its output for auth signals.
|
||||
|
||||
**Error strings the agent should grep:**
|
||||
|
||||
- `"Playwright not installed"` → install `[browser]`
|
||||
- `"rookiepy"` (in stderr of `pip install`) → expected on Python 3.13+; skip `[cookies]` and use interactive `notebooklm login`
|
||||
- `"status": "ok"` (in `auth check --json`) → auth file present and parses; pair with `--test` for network validation
|
||||
|
||||
### B. End user
|
||||
|
||||
Occasional CLI use.
|
||||
|
||||
**Prerequisites:** Python 3.10+ already installed.
|
||||
|
||||
**Recommended — isolated install (macOS / Linux / Windows):**
|
||||
|
||||
<!-- not mirrored: end-user isolated install (pipx / uv tool); CONTRIBUTING.md targets in-repo contributors -->
|
||||
```bash
|
||||
uv tool install "notebooklm-py[browser]"
|
||||
# OR, with pipx:
|
||||
pipx install "notebooklm-py[browser]"
|
||||
```
|
||||
|
||||
Both put `notebooklm` on your PATH in a dedicated environment, so they work even where the system Python is locked down — modern macOS (Homebrew) and Debian/Ubuntu reject a plain `pip install` into it with `error: externally-managed-environment` ([PEP 668](https://peps.python.org/pep-0668/)). (If you don't have `uv` yet: <https://docs.astral.sh/uv/getting-started/installation/>.)
|
||||
|
||||
Plain `pip` is fine **inside a virtualenv**, or on Windows (python.org's Python is not externally-managed):
|
||||
|
||||
<!-- not mirrored: end-user pip install in a venv (Persona B); CONTRIBUTING.md tracks the in-repo `uv sync` flow only -->
|
||||
```bash
|
||||
pip install "notebooklm-py[browser]"
|
||||
```
|
||||
|
||||
**Post-install:** Run `notebooklm login` once. The CLI auto-installs Chromium on first run (~170 MB, 30–90 s, **no progress bar — be patient**).
|
||||
|
||||
**Verify:**
|
||||
|
||||
<!-- not mirrored: end-user post-install verify (Persona B); contributors run the test suite instead -->
|
||||
```bash
|
||||
notebooklm --version
|
||||
notebooklm login # opens Chromium for Google sign-in
|
||||
notebooklm auth check --test # confirms auth roundtrip, with explicit success message
|
||||
```
|
||||
|
||||
### C. Library user
|
||||
|
||||
Embedding `notebooklm-py` in a Python application.
|
||||
|
||||
**Recommended:** `pip install notebooklm-py` (in your app's venv).
|
||||
|
||||
**Post-install:** None for runtime use. To programmatically run interactive login from your app, add `[browser]` and run `playwright install chromium`.
|
||||
|
||||
**Why no extras by default:** all RPC traffic uses `httpx`; auth is cookie-based (`src/notebooklm/auth.py`). Apps can ship a pre-generated `storage_state.json` and never touch Playwright.
|
||||
|
||||
**Verify:**
|
||||
|
||||
```python
|
||||
import notebooklm
|
||||
print(notebooklm.__version__)
|
||||
```
|
||||
|
||||
> **Production deployment patterns (tracked in [#417](https://github.com/teng-lin/notebooklm-py/issues/417)).** Production-grade FastAPI/Django integration — client lifetime in a `lifespan` handler, httpx pool sizing, behavior under concurrent CSRF refresh, multi-tenant `storage_state.json` rotation, a service-shaped Dockerfile, and structured rate-limit/backoff patterns — is not yet covered in `docs/python-api.md`. These were intentionally deferred from the install-docs consolidation (PR #416) to keep its scope focused. See [#417](https://github.com/teng-lin/notebooklm-py/issues/417) for the gap inventory and acceptance criteria.
|
||||
|
||||
### D. Headless server or CI
|
||||
|
||||
**Recommended:** `pip install notebooklm-py`
|
||||
|
||||
**Post-install (3-step recipe — Playwright is *not* required on the server):**
|
||||
|
||||
1. **On a workstation with a display**, install with `[browser]` and log in once:
|
||||
<!-- not mirrored: headless-server bootstrap step 1 (Persona D); not part of contributor flow -->
|
||||
```bash
|
||||
pip install "notebooklm-py[browser]"
|
||||
playwright install chromium
|
||||
notebooklm login # writes ~/.notebooklm/profiles/default/storage_state.json
|
||||
```
|
||||
2. **Move the auth file to the server.** Either ship it as a file:
|
||||
<!-- not mirrored: headless-server bootstrap step 2a (scp); not part of contributor flow -->
|
||||
```bash
|
||||
scp ~/.notebooklm/profiles/default/storage_state.json \
|
||||
user@server:~/.notebooklm/profiles/default/storage_state.json
|
||||
```
|
||||
or stuff the contents into a CI / deployment env var (preferred for ephemeral runners):
|
||||
<!-- not mirrored: headless-server bootstrap step 2b (env var); not part of contributor flow -->
|
||||
```bash
|
||||
export NOTEBOOKLM_AUTH_JSON="$(cat ~/.notebooklm/profiles/default/storage_state.json)"
|
||||
```
|
||||
|
||||
> **CI env-var notes:**
|
||||
> - `storage_state.json` is typically 4–15 KB — well under GitHub Actions' 48 KB single-secret cap.
|
||||
> - Watch for trailing newlines: pipe with `tr -d '\n'` if your secret-set tool adds one (`cat ... | tr -d '\n' | gh secret set NOTEBOOKLM_AUTH_JSON`).
|
||||
> - For **ephemeral runners** (GitHub Actions, GitLab CI — no persistent disk between runs), the layer-5 in-process refresh from [troubleshooting.md](troubleshooting.md#authentication-errors) cannot persist rotated cookies. Run `notebooklm auth refresh` periodically on a workstation cron and push the refreshed file with `gh secret set NOTEBOOKLM_AUTH_JSON < ~/.notebooklm/profiles/default/storage_state.json`.
|
||||
3. **On the server**, run any non-`login` command:
|
||||
<!-- not mirrored: headless-server smoke test; not part of contributor flow -->
|
||||
```bash
|
||||
notebooklm list
|
||||
notebooklm auth check --test # verifies the cookies still authenticate against Google
|
||||
```
|
||||
|
||||
**Why no extras:** reduces the install surface to 4 deps (`httpx`, `click`, `rich`, `filelock`); avoids 200+ MB Chromium download in CI images.
|
||||
|
||||
For runtime configuration (env vars, profiles, parallel agents), see [configuration.md#headless-servers--containers](configuration.md#headless-servers--containers).
|
||||
|
||||
#### Alternative: master-token auth (no cookie file to ship, survives expiry)
|
||||
|
||||
The cookie-copy recipe above ships a `storage_state.json` that eventually
|
||||
expires (cookies are short-lived; ephemeral CI runners can't persist rotations).
|
||||
The **master-token** path instead holds one durable Google master token and
|
||||
**mints fresh web cookies from it on demand** — no browser per session, and an
|
||||
expired session **re-mints automatically** (no manual re-login). One browser
|
||||
sign-in, then headless forever.
|
||||
|
||||
<!-- not mirrored: master-token headless bootstrap (Persona D); not part of contributor flow -->
|
||||
```bash
|
||||
pip install "notebooklm-py[headless]" # adds gpsoauth (pure-Python)
|
||||
|
||||
# One-time bootstrap (a visible browser opens Google's EmbeddedSetup; sign in
|
||||
# with a DEDICATED/throwaway account, and the single-use oauth_token is captured
|
||||
# automatically). Add [browser] for the auto-capture, or paste it with
|
||||
# --oauth-token <value> on a headless box.
|
||||
notebooklm login --master-token --account you@gmail.com
|
||||
|
||||
# Ship master_token.json to the server instead of storage_state.json:
|
||||
scp ~/.notebooklm/profiles/default/master_token.json \
|
||||
user@server:~/.notebooklm/profiles/default/master_token.json
|
||||
|
||||
# On the server, just run commands — cookies are minted/refreshed as needed:
|
||||
notebooklm list
|
||||
# Force a re-mint by hand (or from cron) any time:
|
||||
notebooklm login --master-token-refresh
|
||||
```
|
||||
|
||||
When a `master_token.json` sits beside a profile's `storage_state.json`, an
|
||||
expired session is recovered by re-minting from the master token in-process
|
||||
(after the normal homepage/RotateCookies/headless ladder is exhausted) — so
|
||||
long-lived headless workers self-heal.
|
||||
|
||||
> ⚠️ **Security:** the master token is **full-account, durable, and
|
||||
> infostealer-grade** — a materially larger blast radius than an expiring
|
||||
> `storage_state.json` (it survives password changes until explicitly revoked).
|
||||
> Use a **dedicated/throwaway Google account only**, store it `0600` (the CLI
|
||||
> does), and never commit or log it. This path uses Google's Android auth flow
|
||||
> (`gpsoauth`) and is unofficial/ToS-grey, like the rest of this client. See
|
||||
> [ADR-0023](adr/0023-master-token-headless-auth.md) for the design and rationale.
|
||||
|
||||
### E. Contributor
|
||||
|
||||
Working on this repo.
|
||||
|
||||
**Recommended (respects the checked-in `uv.lock`):**
|
||||
|
||||
<!-- not mirrored: contributor bootstrap with git clone + cd; CONTRIBUTING.md picks up after the clone with the canonical `uv sync --frozen --extra browser --extra dev --extra markdown` command (enforced verbatim by scripts/check_ci_install_parity.py). -->
|
||||
```bash
|
||||
git clone https://github.com/teng-lin/notebooklm-py.git
|
||||
cd notebooklm-py
|
||||
uv sync --frozen --extra browser --extra dev --extra markdown
|
||||
source .venv/bin/activate
|
||||
uv run playwright install chromium
|
||||
pre-commit install
|
||||
```
|
||||
|
||||
**Why `uv sync --frozen` and not `uv pip install -e ".[all]"`:** the repo has a checked-in `uv.lock`. `uv sync --frozen` enforces the lockfile and fails fast on drift; `uv pip install` ignores the lockfile and re-resolves transitively (will silently get newer versions of `playwright`, `ruff`, etc.).
|
||||
|
||||
**Why three extras and not `[all]`:** `[all]` is `pip` extras semantics. `uv sync --extra X` is the `uv` equivalent. The three extras here are the contributor subset of `[all]` = `[browser, dev, markdown, mcp, server]`. `cookies` is intentionally excluded (`rookiepy` build issues on Python 3.13+), and `mcp` / `server` are omitted from the default contributor flow because those adapters are not needed for the standard local suite; opt in via `--extra cookies` / `--extra mcp` / `--extra server` if needed.
|
||||
|
||||
**Why `browser` is part of the contributor install:** the default local test suite includes unit tests that import and patch `playwright.sync_api`, even though they do not launch a real browser. `uv sync --frozen --extra dev` installs pytest/ruff/mypy but not Playwright, so `uv run pytest` will fail with `ModuleNotFoundError: No module named 'playwright'`. Use the full contributor command above before running the default test suite.
|
||||
|
||||
**Linux only:** `uv run playwright install-deps chromium` (scoped form, matches `test.yml`).
|
||||
|
||||
**Pre-commit checklist (run before every commit):**
|
||||
|
||||
```bash
|
||||
uv run ruff format --check . && \
|
||||
uv run ruff check . && \
|
||||
uv run mypy src/notebooklm --ignore-missing-imports && \
|
||||
uv run pytest --cov=src/notebooklm --cov-report=term-missing --cov-fail-under=90
|
||||
```
|
||||
|
||||
**Verify:**
|
||||
|
||||
<!-- not mirrored: contributor verify block; CONTRIBUTING.md mirrors only the pre-commit checklist (the more frequent, per-commit version) -->
|
||||
```bash
|
||||
notebooklm --version
|
||||
uv run pytest --cov=src/notebooklm --cov-report=term-missing --cov-fail-under=90
|
||||
uv run pre-commit run --all-files
|
||||
```
|
||||
|
||||
### F. Power user
|
||||
|
||||
Non-default browsers, cookie extraction, markdown source dumps.
|
||||
|
||||
> **Why this section uses the combined `[browser,cookies]` form** — unlike Persona A, which uses two separate `pip install` calls so a `rookiepy` build failure doesn't leave the user with nothing: power users explicitly opted in, know what `rookiepy` is, and prefer the all-or-nothing tradeoff (single command, no wrapping logic).
|
||||
|
||||
> ⚠️ **Don't use `[all]` for power-user setups.** `[all]` deliberately *excludes* `cookies` (see [§ All vs All-Extras](#all-vs-all-extras)). If you `pip install "notebooklm-py[all]"` and then try `--browser-cookies`, you'll get an opaque `rookiepy` import error. For everything-and-the-kitchen-sink, use `pip install "notebooklm-py[browser,cookies,markdown]"` explicitly (Python ≤ 3.12 only).
|
||||
|
||||
- **`--browser-cookies` (no Playwright login):** `pip install "notebooklm-py[browser,cookies]"`. **Caveat:** `rookiepy` may fail to install on Python 3.13/3.14; use Python 3.12 or accept the risk. See [cli-reference.md#authentication-login](cli-reference.md#authentication-login) for the full `--browser-cookies` syntax, including `chrome::<profile-name-or-directory>` for one Chromium user-profile and `firefox::<container>` for Firefox Multi-Account Containers (on every OS — not just macOS). Use `notebooklm auth inspect --browser <browser>` for previewing available accounts before import.
|
||||
- **Markdown source dumps:** `pip install "notebooklm-py[markdown]"` for `notebooklm source fulltext -f markdown`.
|
||||
- **Edge instead of Chromium:** install Microsoft Edge from [microsoft.com/edge](https://www.microsoft.com/edge) first — `--browser msedge` does NOT auto-install Edge (only `--browser chromium` auto-installs). Then `notebooklm login --browser msedge`.
|
||||
- **Multi-account (personal + work):** see [configuration.md#multiple-accounts](configuration.md#multiple-accounts). Common power-user flow: `notebooklm profile create work && notebooklm -p work login --browser-cookies edge --account work@corp.com`. Use `--all-accounts` to bootstrap profiles for every signed-in Google account in one command.
|
||||
|
||||
---
|
||||
|
||||
## Optional extras matrix
|
||||
|
||||
Source of truth: `pyproject.toml` `[project.optional-dependencies]`.
|
||||
|
||||
| Extra | What it adds | When you need it | pip command | uv (in your project) |
|
||||
|---|---|---|---|---|
|
||||
| (none) | `httpx`, `click`, `rich`, `filelock` | All RPC operations, all CLI commands except `login`. Suffices when you ship a `storage_state.json`. | `pip install notebooklm-py` | `uv add notebooklm-py` |
|
||||
| `browser` | `playwright>=1.40.0` | `notebooklm login` (interactive). | `pip install "notebooklm-py[browser]"` | `uv add "notebooklm-py[browser]"` |
|
||||
| `cookies` | `rookiepy>=0.1.0` | `notebooklm login --browser-cookies <browser>`, `notebooklm auth inspect`. | `pip install "notebooklm-py[cookies]"` | `uv add "notebooklm-py[cookies]"` |
|
||||
| `headless` | `gpsoauth>=1.1.0` | `notebooklm login --master-token` — headless auth that mints/refreshes web cookies from a durable master token, no per-session browser. Pure-Python (in `all`). See [§ D](#d-headless-server-or-ci). | `pip install "notebooklm-py[headless]"` | `uv add "notebooklm-py[headless]"` |
|
||||
| `impersonate` | `curl_cffi>=0.11` | **Experimental.** Browser TLS/JA3 impersonation transport — set `NOTEBOOKLM_TRANSPORT=curl_cffi` to route the authenticated API surface through a Chrome-fingerprinted connection (insurance vs TLS fingerprint-gating); override the profile with `NOTEBOOKLM_IMPERSONATE` (default `chrome`, e.g. `safari`, `chrome131`). Native wheels. | `pip install "notebooklm-py[impersonate]"` | `uv add "notebooklm-py[impersonate]"` |
|
||||
| `markdown` | `markdownify>=0.14.1` | `notebooklm source fulltext -f markdown`. | `pip install "notebooklm-py[markdown]"` | `uv add "notebooklm-py[markdown]"` |
|
||||
| `mcp` | `fastmcp>=2.14` | Run the MCP server (`notebooklm-mcp`) so an MCP client/agent can drive NotebookLM as tools. | `pip install "notebooklm-py[mcp]"` | `uv add "notebooklm-py[mcp]"` |
|
||||
| `server` | `fastapi`, `uvicorn[standard]`, `python-multipart` | The localhost REST API server (`notebooklm-server`, experimental). See [§ REST API server](#rest-api-server). | `pip install "notebooklm-py[server]"` | `uv add "notebooklm-py[server]"` |
|
||||
| `dev` | pytest stack, mypy, ruff (`==0.15.15` exact pin), pre-commit (`>=4.5.1`), vcrpy | Contributor tooling only. Not sufficient for this repo's default `uv run pytest`; add `browser` too because some unit tests import Playwright. | `pip install "notebooklm-py[dev]"` | `uv add "notebooklm-py[dev]"` (in your project) — but contributors *to this repo* use the [Persona E](#e-contributor) `uv sync` flow instead |
|
||||
| `all` | Resolves to `browser` + `dev` + `headless` + `markdown` + `mcp` + `server` (**not `cookies`**) | Contributors who do not need `rookiepy`. | `pip install "notebooklm-py[all]"` | `uv add "notebooklm-py[all]"` (in your project) — see [All vs All-Extras](#all-vs-all-extras) |
|
||||
|
||||
> **Note on `uv` columns:** the `uv (in your project)` column is for users adding `notebooklm-py` as a dependency in **their own** project (requires a `pyproject.toml` in that project). Contributors working inside *this* repo use the Persona E flow (`uv sync --frozen --extra ...`), governed by this repo's `uv.lock`. Do not run `uv sync` outside a project — it errors with `No pyproject.toml found`.
|
||||
|
||||
---
|
||||
|
||||
## REST API server
|
||||
|
||||
> **⚠️ Experimental.** Like the MCP adapter, the REST server is experimental: the `/v1` surface and behavior may change in a minor release, and it is excluded from the public-API compatibility gate. Pin a version before relying on it for automation. The server also logs an experimental warning on every startup.
|
||||
|
||||
A single-tenant, localhost REST API over the same transport-neutral core as the CLI — the natural shape for scripting and agent automation (feed a notebook, generate an artifact, pull it down) without spawning a CLI process per call.
|
||||
|
||||
<!-- not mirrored: the server extra is end-user/automation tooling, not part of the contributor `uv sync` flow; CONTRIBUTING.md tracks only browser/dev/markdown. -->
|
||||
```bash
|
||||
uv tool install "notebooklm-py[server]" # fastapi + uvicorn + python-multipart
|
||||
# OR, with pipx: pipx install "notebooklm-py[server]" (or plain pip inside a venv)
|
||||
```
|
||||
|
||||
**Prerequisite:** a provisioned account (`storage_state.json`) from `notebooklm login`. The server holds one account for the process; it does not run browser login itself.
|
||||
|
||||
**Launch:**
|
||||
|
||||
<!-- not mirrored: REST-server launch (end-user/automation tooling); CONTRIBUTING.md tracks only the contributor `uv sync` flow. -->
|
||||
```bash
|
||||
export NOTEBOOKLM_SERVER_TOKEN="$(openssl rand -hex 32)" # REQUIRED — the server refuses to start without it
|
||||
notebooklm-server --host 127.0.0.1 --port 8000 # loopback-only by default
|
||||
```
|
||||
|
||||
Configuration is read from `NOTEBOOKLM_SERVER_*` env vars (overridable by the matching flags):
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `NOTEBOOKLM_SERVER_TOKEN` | *(unset)* | Bearer token every request must present. **Required** — fail-closed if unset. |
|
||||
| `NOTEBOOKLM_SERVER_HOST` | `127.0.0.1` | Bind host. Non-loopback is refused unless the elevated-risk override below is set. |
|
||||
| `NOTEBOOKLM_SERVER_PORT` | `8000` | Bind port. |
|
||||
| `NOTEBOOKLM_SERVER_ALLOW_EXTERNAL_BIND` | *(unset)* | ⚠️ Set to `1` to bind a non-loopback interface. Only behind a trusted reverse proxy — this exposes account-fronting credentials to the network. |
|
||||
| `NOTEBOOKLM_SERVER_SOURCE_MUTATION_CONCURRENCY` | `4` | Max concurrent source create/rename/delete/batch handlers. |
|
||||
| `NOTEBOOKLM_SERVER_SOURCE_WAIT_CONCURRENCY` | `4` | Max concurrent source wait handlers. |
|
||||
| `NOTEBOOKLM_SERVER_GENERATION_CONCURRENCY` | `2` | Max concurrent artifact generation/retry handlers. |
|
||||
| `NOTEBOOKLM_SERVER_DOWNLOAD_CONCURRENCY` | `2` | Max concurrent artifact download handlers. |
|
||||
| `NOTEBOOKLM_SERVER_RESEARCH_CONCURRENCY` | `2` | Max concurrent research start/cancel/import handlers. |
|
||||
| `NOTEBOOKLM_SERVER_CHAT_CONCURRENCY` | `4` | Max concurrent blocking chat ask handlers. |
|
||||
|
||||
The concurrency knobs are route-group backpressure for expensive work. They do not gate `/healthz` or cheap read/list/poll routes.
|
||||
|
||||
**Surface:** every route is under `/v1` and requires `Authorization: Bearer <token>` plus a loopback `Host` header (a DNS-rebinding guard). `/healthz` is the one public, token-less route. The auto-generated `/docs` / `/openapi.json` schema UI is disabled (it would otherwise be reachable token-less).
|
||||
|
||||
<!-- not mirrored: REST-server curl examples (end-user/automation tooling); not part of the contributor install flow. -->
|
||||
```bash
|
||||
TOKEN=$NOTEBOOKLM_SERVER_TOKEN
|
||||
BASE=http://127.0.0.1:8000
|
||||
|
||||
curl $BASE/healthz # {"ok": true} (no token)
|
||||
curl -H "Authorization: Bearer $TOKEN" $BASE/v1/notebooks # list notebooks
|
||||
curl -H "Authorization: Bearer $TOKEN" -d '{"title":"My NB"}' \
|
||||
-H 'Content-Type: application/json' $BASE/v1/notebooks # create
|
||||
curl -H "Authorization: Bearer $TOKEN" -d '{"url":"https://example.com"}' \
|
||||
-H 'Content-Type: application/json' $BASE/v1/notebooks/<id>/sources/url
|
||||
curl -H "Authorization: Bearer $TOKEN" -d '{"question":"Summarize"}' \
|
||||
-H 'Content-Type: application/json' $BASE/v1/notebooks/<id>/chat # blocking answer
|
||||
curl -H "Authorization: Bearer $TOKEN" $BASE/v1/notebooks/<id>/share # sharing status
|
||||
```
|
||||
|
||||
Endpoints: `/v1/notebooks` (list/get/create/delete); `/v1/notebooks/{id}/sources` (list/get/add via `url`·`text`·`file`/delete); `/v1/notebooks/{id}/notes` (list/get/create/update via `PUT`/delete); `/v1/notebooks/{id}/chat` (blocking ask, no streaming); `/v1/notebooks/{id}/artifacts` (list / generate / poll / download); `/v1/notebooks/{id}/share` (status / public link / users / view level). Long-running work (source ingest, artifact generation) is **poll-the-resource**: the create call returns immediately and the matching `GET` reports `pending` until the resource is ready (`200`), `404` for an id the server never created, `409`/`410` for a failed/removed artifact.
|
||||
|
||||
**Artifacts & uploads:**
|
||||
|
||||
<!-- not mirrored: REST-server artifact/upload curl examples (end-user/automation tooling); not part of the contributor install flow. -->
|
||||
```bash
|
||||
# Generate (non-blocking → 202 + task_id). Omit source_ids to use ALL sources
|
||||
# (like the CLI); pass them to scope. Some types (quiz/flashcards) need at least one source.
|
||||
curl -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
|
||||
-d '{"type":"quiz"}' $BASE/v1/notebooks/<id>/artifacts # → {"task_id": ...}
|
||||
curl -H "Authorization: Bearer $TOKEN" $BASE/v1/notebooks/<id>/artifacts/<task_id> # poll
|
||||
curl -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
|
||||
-d '{"type":"audio"}' $BASE/v1/notebooks/<id>/artifacts/download -o out.mp3 # download
|
||||
# File upload is multipart (the original filename + content-type are preserved):
|
||||
curl -H "Authorization: Bearer $TOKEN" -F 'file=@./notes.pdf' \
|
||||
$BASE/v1/notebooks/<id>/sources/file
|
||||
```
|
||||
|
||||
**Error envelope:** every failure is `{"error": {"category": "...", "message": "..."}}` with a category-derived HTTP status — `not_found`→404, `validation`→400/422, `auth`→401/403, `rate_limited`→429, `notebook_limit`→409, server/network→502, timeouts→504. The category is classified once by `_app.errors.classify`, shared with the CLI.
|
||||
|
||||
---
|
||||
|
||||
## Post-install steps
|
||||
|
||||
### `playwright install chromium` — when required, when auto-installed
|
||||
|
||||
- **Required**: when you'll use `notebooklm login` (the interactive Playwright flow), unless the CLI auto-installs Chromium for you (it does — see `ensure_chromium_installed()` in `cli/services/playwright_login.py`, which runs `python -m playwright install chromium` on first login if Chromium is missing).
|
||||
- **Not required**: for headless servers (Persona D), library use (Persona C), or `--browser-cookies`-based auth (Persona A/F with `[cookies]`).
|
||||
|
||||
### `playwright install-deps chromium` — Linux system libraries
|
||||
|
||||
On Debian/Ubuntu, Playwright needs system libs for Chromium. Run after `playwright install chromium`:
|
||||
|
||||
<!-- not mirrored: Linux-specific Playwright system-library install; CI runs `uv run playwright install-deps chromium` directly in test.yml -->
|
||||
```bash
|
||||
playwright install-deps chromium # scoped to chromium; matches CI
|
||||
```
|
||||
|
||||
Works without `sudo` if you're root or have passwordless sudo. Otherwise `sudo playwright install-deps chromium`.
|
||||
|
||||
### First-time `notebooklm login`
|
||||
|
||||
<!-- not mirrored: end-user first-login walkthrough; contributors typically reuse an existing storage_state.json -->
|
||||
```bash
|
||||
notebooklm login # opens Chromium for Google sign-in
|
||||
notebooklm auth check --test # verify
|
||||
```
|
||||
|
||||
The login command:
|
||||
- Auto-installs Chromium if missing (Persona A/B/E).
|
||||
- Saves cookies to `~/.notebooklm/profiles/<profile>/storage_state.json`.
|
||||
- Uses a *persistent* browser profile so subsequent logins are faster.
|
||||
|
||||
### `notebooklm skill install` — for AI agents (Persona A)
|
||||
|
||||
Registers the skill into local agent skill directories:
|
||||
|
||||
<!-- not mirrored: agent skill directory registration; out of scope for the contributor README -->
|
||||
```bash
|
||||
notebooklm skill install # writes ~/.claude/skills/notebooklm/, ~/.agents/skills/notebooklm/
|
||||
```
|
||||
|
||||
Optional — only needed if your agent harness reads from those directories and the skill isn't already present.
|
||||
|
||||
### Running the MCP server (`mcp` extra)
|
||||
|
||||
The MCP server ships behind the optional `mcp` extra (see the extras matrix above) and exposes the same `_app/` business logic over the Model Context Protocol.
|
||||
|
||||
<!-- not mirrored: end-user MCP run/config pointer; out of scope for the contributor README -->
|
||||
```bash
|
||||
notebooklm-mcp # installed console script (stdio transport)
|
||||
uvx --from "notebooklm-py[mcp]" notebooklm-mcp # no install — run straight from PyPI
|
||||
```
|
||||
|
||||
Wire it into an MCP client with either:
|
||||
- `notebooklm mcp install <client>` — auto-writes the server config for `claude-desktop`, `claude-code`, `cursor`, or `windsurf`; or
|
||||
- the one-click `.mcpb` desktop bundle — download it from the [latest release](https://github.com/teng-lin/notebooklm-py/releases/latest) (**Assets**) and use Claude Desktop's "Install Extension". Each stable release attaches a prebuilt, version-matched bundle; see [`desktop-extension/README.md`](../desktop-extension/README.md).
|
||||
|
||||
Full usage walkthrough (auth, transports, the 35 tools, workflows, troubleshooting): **[mcp-guide.md](mcp-guide.md)**.
|
||||
|
||||
---
|
||||
|
||||
## Verifying your install
|
||||
|
||||
| Command | What it checks | Use when |
|
||||
|---|---|---|
|
||||
| `notebooklm --version` | Package installed correctly. | Always. |
|
||||
| `notebooklm auth check --json` | Auth file parses; `SID` cookie present. Returns `{"status": "ok"\|"error", "checks": {...}}`. | Agents (machine-parseable). |
|
||||
| `notebooklm auth check --test` | Same + network token-fetch validates that cookies still authenticate against Google. | End users (after login). |
|
||||
| `notebooklm auth check --test --json` | Both. | Agents that need to confirm the cookies aren't stale. |
|
||||
| `notebooklm list` | Package + auth + RPC roundtrip all work. | After login, as a smoke test. |
|
||||
|
||||
> **Important:** `notebooklm status` reports *context state* (which notebook is selected). It is **not** an auth check. See [Common gotchas](#common-gotchas-appendix).
|
||||
|
||||
**Your first end-to-end run:**
|
||||
|
||||
<!-- not mirrored: end-user smoke test; contributors run `uv run pytest` instead -->
|
||||
```bash
|
||||
notebooklm create "My First Notebook"
|
||||
notebooklm source add 'https://en.wikipedia.org/wiki/Python_(programming_language)'
|
||||
notebooklm ask "Summarize the sources in three sentences"
|
||||
```
|
||||
|
||||
For the full CLI surface, see [cli-reference.md](cli-reference.md).
|
||||
|
||||
---
|
||||
|
||||
## Platform notes
|
||||
|
||||
| Platform | Install-time notes | Diagnostic detail |
|
||||
|---|---|---|
|
||||
| **macOS** | Chromium auto-downloads on first login. `--browser-cookies` from Chrome/Edge/Brave/Opera may prompt for Keychain access. | [troubleshooting.md#macos](troubleshooting.md#macos) |
|
||||
| **Linux** | (a) `playwright install-deps chromium` for system libs (Debian/Ubuntu). (b) **Known bug:** `playwright > 1.57` may fail with `TypeError: onExit is not a function` — pin `playwright==1.57.0`. | [troubleshooting.md#linux](troubleshooting.md#linux) |
|
||||
| **Windows** | The library auto-configures `WindowsSelectorEventLoopPolicy` and `PYTHONUTF8=1`. Prefer plain `pip install` (uv/pipx less common on Windows). | [troubleshooting.md#windows](troubleshooting.md#windows) |
|
||||
| **WSL** | The browser opens in the Windows host (expected); `storage_state.json` lives in the WSL filesystem. | [troubleshooting.md#wsl](troubleshooting.md#wsl) |
|
||||
|
||||
---
|
||||
|
||||
## Upgrading and uninstalling
|
||||
|
||||
<!-- not mirrored: end-user upgrade commands; contributors `git pull && uv sync --frozen ...` instead -->
|
||||
```bash
|
||||
pip install --upgrade notebooklm-py # latest patch
|
||||
pip install --upgrade "notebooklm-py[browser]" # preserves your extras
|
||||
```
|
||||
|
||||
For pinning patterns and version-stability guarantees, see [stability.md](stability.md).
|
||||
|
||||
To uninstall:
|
||||
|
||||
<!-- not mirrored: end-user uninstall; contributors `git clean -fdx` and remove the worktree instead -->
|
||||
```bash
|
||||
pip uninstall notebooklm-py
|
||||
rm -rf ~/.notebooklm # optional: remove auth state
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common gotchas (appendix)
|
||||
|
||||
### All vs All-Extras
|
||||
|
||||
> ⚠️ **`pip install ".[all]"` and `uv sync --all-extras` are not equivalent.**
|
||||
>
|
||||
> - `pyproject.toml` defines: `all = ["notebooklm-py[browser,dev,markdown,mcp,server]"]` — a self-referential extras string that resolves to **browser + dev + markdown + mcp + server only**. It deliberately excludes `cookies` because `rookiepy` has install issues on Python 3.13+ ([CHANGELOG `[0.4.1]`](../CHANGELOG.md)).
|
||||
> - `uv sync --all-extras` installs **every** extra including `cookies`, and may fail on Python 3.13/3.14.
|
||||
> - In this repo, prefer `uv sync --frozen --extra browser --extra dev --extra markdown`.
|
||||
|
||||
### `uv pip install` vs `uv sync`
|
||||
|
||||
- `uv pip install -e ".[all]"` ignores the checked-in `uv.lock` — it re-resolves dependencies and may pull newer versions of `playwright`, `ruff`, etc. than the lock specifies.
|
||||
- `uv sync --frozen` enforces the lockfile and fails fast on drift. **This is what contributors should use.**
|
||||
- `uv sync` (no `--frozen`) silently updates `uv.lock` if `pyproject.toml` has changed. Use only when intentionally bumping deps.
|
||||
|
||||
### `notebooklm status` ≠ auth
|
||||
|
||||
`notebooklm status` reports the *currently selected notebook* (context). It does NOT report whether you are authenticated. For auth, use `notebooklm auth check` (or `--json` / `--test --json` for machine output and network validation).
|
||||
@@ -0,0 +1,457 @@
|
||||
# MCP server guide
|
||||
|
||||
> **Experimental / preview.** The MCP server ships behind the optional `mcp` extra. Its
|
||||
> tool surface (names, parameters, output shapes) is **not** covered by the library's semver
|
||||
> guarantees and may change between releases. `pip install notebooklm-py` is unaffected — the
|
||||
> server and its dependencies only arrive with the `mcp` extra.
|
||||
|
||||
The MCP server exposes NotebookLM to any [Model Context Protocol](https://modelcontextprotocol.io)
|
||||
client (Claude Desktop, Claude Code, Cursor, Windsurf, …) as a set of **32 tools** — manage
|
||||
notebooks and sources, chat over a notebook's sources, generate and download studio artifacts,
|
||||
and run deep research. It is a thin adapter over the same business logic the CLI uses, so it
|
||||
behaves identically to `notebooklm <command>`.
|
||||
|
||||
## Install
|
||||
|
||||
The server is behind the `mcp` extra (pulls in `fastmcp`):
|
||||
|
||||
```bash
|
||||
pip install "notebooklm-py[mcp]"
|
||||
# or run with no install, straight from PyPI:
|
||||
uvx --from "notebooklm-py[mcp]" notebooklm-mcp --help
|
||||
```
|
||||
|
||||
## Authenticate (once)
|
||||
|
||||
The server reuses the CLI's stored credentials — it does **not** log in on its own. Authenticate
|
||||
once before starting it:
|
||||
|
||||
```bash
|
||||
notebooklm login
|
||||
# or, if you didn't install the package:
|
||||
uvx --from "notebooklm-py[mcp]" notebooklm login
|
||||
```
|
||||
|
||||
Credentials are stored per profile under `~/.notebooklm/`. The server binds the **active profile**
|
||||
at startup (override with `--profile`, below). See [configuration.md](configuration.md) for profiles
|
||||
and multi-account setup.
|
||||
|
||||
## Connect a client
|
||||
|
||||
The fastest path is the auto-config command, which writes the server block into a client's MCP
|
||||
config (idempotent, never clobbers other servers):
|
||||
|
||||
```bash
|
||||
notebooklm mcp install claude-desktop # or: claude-code | cursor | windsurf
|
||||
```
|
||||
|
||||
| Client | Config written |
|
||||
|--------|----------------|
|
||||
| `claude-desktop` | `claude_desktop_config.json` (per-OS location) |
|
||||
| `claude-code` | `~/.claude.json` (user scope) |
|
||||
| `cursor` | `~/.cursor/mcp.json` |
|
||||
| `windsurf` | `~/.codeium/windsurf/mcp_config.json` |
|
||||
|
||||
It writes a block that launches the server via `uvx` (so only `uv` needs to be on the host):
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"mcpServers": {
|
||||
"notebooklm": {
|
||||
"command": "uvx",
|
||||
"args": ["--from", "notebooklm-py[mcp]", "notebooklm-mcp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Restart the client after installing. For a one-click Claude Desktop bundle,
|
||||
download `notebooklm-mcp.mcpb` from the
|
||||
[latest release](https://github.com/teng-lin/notebooklm-py/releases/latest)
|
||||
(**Assets**) and use "Install Extension"; see
|
||||
[`desktop-extension/README.md`](../desktop-extension/README.md) for details.
|
||||
|
||||
## Run it directly
|
||||
|
||||
The console script is `notebooklm-mcp`:
|
||||
|
||||
```bash
|
||||
notebooklm-mcp # stdio transport (default — for desktop hosts)
|
||||
notebooklm-mcp --profile work # bind a specific auth profile
|
||||
notebooklm-mcp --transport http # loopback streamable-HTTP on 127.0.0.1:9420
|
||||
notebooklm-mcp --transport http --port 9000
|
||||
```
|
||||
|
||||
| Flag | Default | Notes |
|
||||
|------|---------|-------|
|
||||
| `--profile` | active profile | which stored auth profile the process binds |
|
||||
| `--transport` | `stdio` | `stdio` (subprocess hosts) or `http` (loopback) |
|
||||
| `--host` | `127.0.0.1` | http only; non-loopback is **refused** unless `NOTEBOOKLM_MCP_ALLOW_EXTERNAL_BIND=1` |
|
||||
| `--port` | `9420` | http only |
|
||||
| `--log-level` | `INFO` | logs go to **stderr**; stdout stays pure JSON-RPC |
|
||||
|
||||
There is no `--token` flag — the HTTP bearer token is **env-only**
|
||||
(`NOTEBOOKLM_MCP_TOKEN`) so it cannot leak via `ps aux`.
|
||||
|
||||
`stdio` is right for Claude Desktop/Code, Cursor, and Windsurf (they launch the server as a
|
||||
subprocess). Use `http` for a local web client or to share one running server across clients on
|
||||
the same machine. The HTTP transport is loopback-only by default; binding to a non-loopback
|
||||
address requires **both** the explicit `NOTEBOOKLM_MCP_ALLOW_EXTERNAL_BIND=1` override **and** a
|
||||
`NOTEBOOKLM_MCP_TOKEN` — the server fails closed (refuses to start) on a network bind without a
|
||||
token, since it fronts a full Google account.
|
||||
|
||||
## Remote deployment (Docker + a tunnel)
|
||||
|
||||
Because master-token auth keeps the session alive unattended (no browser), the HTTP transport can
|
||||
run as a **remote connector** reachable from Claude Code, Claude Desktop, claude.ai, mobile, and ChatGPT.
|
||||
The [`deploy/`](../deploy/) directory ships a turn-key Docker + Compose stack with a **tunnel
|
||||
sidecar** — pick one via a Compose profile — so you get HTTPS with **no public IP, no open ports,
|
||||
and no TLS certificate to manage** (the tunnel terminates TLS at its edge).
|
||||
|
||||
**→ The full step-by-step lives in [`deploy/README.md`](../deploy/README.md)** — run it from
|
||||
inside `deploy/` (`make setup` → finish the one manual tunnel step → `make up`), where the Compose
|
||||
stack, `Makefile`, and `.env.example` it references sit. It walks both tunnels end to end:
|
||||
**Cloudflare** (needs a domain in your Cloudflare account) and **Tailscale Funnel** (no domain — a
|
||||
free, stable `*.ts.net` HTTPS hostname). The rest of this section is the two things worth knowing
|
||||
before you start: the auth model and remote file transfer.
|
||||
|
||||
**Two auth methods coexist on one `/mcp`** (FastMCP `MultiAuth`):
|
||||
- **Claude Code / Desktop** → the static `NOTEBOOKLM_MCP_TOKEN` bearer (an `Authorization` header).
|
||||
- **claude.ai (web/mobile) and ChatGPT** (Developer Mode) → optional **self-hosted OAuth** (one
|
||||
password, no external IdP): set `NOTEBOOKLM_MCP_OAUTH_PASSWORD` (≥16 random chars) +
|
||||
`NOTEBOOKLM_MCP_OAUTH_BASE_URL` (the **bare public origin**, no `/mcp`). Both connector UIs are
|
||||
OAuth-only (no bearer field). Unset → bearer-only (Claude Code/Desktop still work).
|
||||
|
||||
Use a **dedicated/throwaway Google account** — the mounted `master_token.json` is a durable
|
||||
full-account credential. Multi-tenant hosting is out of scope for this single-tenant setup.
|
||||
|
||||
### File upload & download (remote)
|
||||
|
||||
The MCP/JSON-RPC channel can't carry large binaries, so over a remote connector
|
||||
`source_add type=file` and `studio_download` broker a **short-lived signed URL**
|
||||
served by the same container; your browser does the byte transfer (see
|
||||
[ADR-0024](adr/0024-mcp-remote-file-transfer.md)). This is the standard pattern for
|
||||
remote MCP file transfer — MCP has no native file-upload primitive, and its native
|
||||
download (binary Resources) is capped far below a podcast/video. (A **small** file
|
||||
can skip the signed URL entirely — see `source_upload_bytes` below.)
|
||||
|
||||
**Enable it:** set `NOTEBOOKLM_MCP_PUBLIC_URL` to your bare public origin (the same
|
||||
host as the tunnel, no `/mcp`). It falls back to `NOTEBOOKLM_MCP_OAUTH_BASE_URL`, so
|
||||
if you configured claude.ai OAuth above, **file transfer is already on**. Unset on a
|
||||
bearer-only deploy → the two file tools return a clear "not configured" error
|
||||
(everything else still works; the server does not refuse to start).
|
||||
|
||||
- **Upload a local file:** `source_add type=file` returns an `upload_required` link.
|
||||
Open it in your browser, pick the file, and it's added to the notebook. (Claude can
|
||||
also `PUT` a file it already holds to that link from its **code-execution sandbox** —
|
||||
but that requires Code Execution enabled **and your server domain whitelisted** in
|
||||
claude.ai Settings → Capabilities → additional allowed domains, or the `PUT` fails.)
|
||||
- **Hand a small file's bytes in-channel (no signed URL):** when an agent holds the
|
||||
bytes but can complete *neither* upload path — e.g. its egress is blocked, so the
|
||||
`agent_upload` POST fails, and no human device has the file — `source_upload_bytes`
|
||||
takes the file as base64 (≤ 10,000 chars, ≈ 7 KB) and the connector adds it
|
||||
server-side, returning the source directly. It works on any transport and needs no
|
||||
`NOTEBOOKLM_MCP_PUBLIC_URL`; a larger file must use the `source_add type=file`
|
||||
signed-URL flow above.
|
||||
- **Download an artifact:** `studio_download` returns a `download_ready` link (a
|
||||
clickable `resource_link`); open it to stream the podcast/video/PDF to your device.
|
||||
The `download_ready` payload is self-describing so a client can render a download
|
||||
affordance *before* opening the URL: alongside `url` / `expires_at` / `artifact_type`
|
||||
/ `artifact_id` it carries `filename` (the artifact title — or the type name on a
|
||||
latest-by-type download — plus the format-resolved extension), `mime_type` (from a
|
||||
central per-type/format table the `/files/dl` route serves with, so the advertised
|
||||
type and the streamed `Content-Type` can't drift), and `size_bytes` (`null` — the
|
||||
size isn't known without eagerly fetching the artifact, which the broker won't do;
|
||||
the route sets the real `Content-Length` when the link is opened).
|
||||
- Links are HMAC-signed and short-lived (upload 15 min, download 30 min) and expire on
|
||||
a server restart. Google Drive (`source_add` with a Drive id) remains a no-browser
|
||||
alternative for adding files. stdio (local) installs are unchanged — they still read
|
||||
and write real local paths directly.
|
||||
|
||||
## Core concepts
|
||||
|
||||
These conventions hold across every tool:
|
||||
|
||||
- **JSON by default.** Read/wait tools, including `source_read`, `source_wait`, and
|
||||
`source_add_and_wait`, return a JSON text content block plus the same
|
||||
`structured_content`. A `resource_link` appears only when a tool explicitly brokers
|
||||
file transfer, such as `studio_download`.
|
||||
- **Name *or* ID.** Every `notebook`/`source`/`note`/`artifact` argument accepts a human title **or**
|
||||
an ID. Both resolve by prefix: an exact title wins, otherwise a **unique title prefix** matches
|
||||
(so `"Scientific"` finds `"Scientific PDF Parsing — …"`), and likewise a full ID or a unique ID
|
||||
prefix. Use the matching `*_list` tool to discover them. An ambiguous name or prefix returns a
|
||||
`VALIDATION` error listing the candidates so you can retry with an exact title or ID. When a name
|
||||
lookup *fails* but is close to a real title — a punctuation-only slip such as a hyphen typed for an
|
||||
em-dash (`—`) or a normal space for a non-breaking one — the error's `Did you mean: …` hint names
|
||||
up to three near-miss candidates, each with its **title and id** inline, so you can retry with the
|
||||
full title or id instead of guessing (a label near-miss reached via `source_list(label=…)` gets the
|
||||
same enrichment on its `VALIDATION` error).
|
||||
- **Canonical IDs come back.** Every response echoes the canonical `notebook_id` (and, where a
|
||||
tool resolves them, the `source_ids` scope / `artifact_id`) — so a call made by *name* hands you
|
||||
the id to chain the next call on.
|
||||
- **Strict IDs-only mode (opt-in).** Set `NOTEBOOKLM_MCP_STRICT_IDS=1` on the server to require a
|
||||
**full canonical id** for every `notebook`/`source`/`note`/`artifact` reference: names, titles, and
|
||||
short id prefixes are rejected with a `VALIDATION` error *before* any list call. This trades the
|
||||
convenience above for deterministic, fail-fast behavior in long-lived automation, where a prefix or
|
||||
title that is unique today can quietly resolve to a different (or ambiguous) entity tomorrow. Off by
|
||||
default. (Governs every notebook/source/note/artifact reference — including studio `item` and
|
||||
`studio_download`'s `artifact_id`; the `source_list(label=…)` name filter is out of scope.)
|
||||
- **Destructive tools need confirmation.** `notebook_delete`, `source_delete`,
|
||||
`studio_delete`, and `share_remove_user` take `confirm` (default `false`). Called without it, they return a `needs_confirmation` preview
|
||||
(with the resolved title) and delete **nothing**; call again with `confirm=true` to execute.
|
||||
- **Sharing-widening tools need confirmation too.** `share_set_user` (every grant/regrade) and
|
||||
`share_set_access` when it would widen link access (`public=true` on a currently-restricted
|
||||
notebook) take `confirm` (default `false`) and return a `needs_confirmation` preview instead of
|
||||
mutating; call again with `confirm=true` to apply. Restricting (`public=false`) and
|
||||
`view_level`-only changes are not gated. These tools are *not* flagged `destructiveHint` — the
|
||||
gate is on the widening direction only.
|
||||
- **Long-running work is non-blocking.** `studio_generate` returns immediately with a `task_id`;
|
||||
poll `studio_status` until it's complete, then `studio_download`. Research is the same shape:
|
||||
`research_start` → `research_status` → `research_import`.
|
||||
- **Mutation envelope.** Synchronous create/rename/update/delete tools return a top-level
|
||||
`status` string naming the outcome — one of `created`, `renamed`, `updated`, `deleted`,
|
||||
`removed`, `added`, `imported`, `cancel_requested`, `configured` (plus the `needs_confirmation` /
|
||||
`upload_required` / `download_ready` flow values) — alongside the affected id(s). An agent can
|
||||
branch on `status` uniformly instead of learning a different success shape per tool. Two
|
||||
carve-outs: the **long-running starters** `studio_generate` / `research_start` return a
|
||||
`task_id` (the handle *is* the result — poll it) rather than a mutation status — as does the
|
||||
re-runner `studio_retry` (its `task_id` equals the artifact id); and the **read** tools
|
||||
`studio_status` / `research_status` key `status` to a lifecycle vocabulary
|
||||
(`in_progress` / `completed` / …), a *different* enum. (Batch
|
||||
`source_add` reports `added` once ≥1 succeeded, `error` if all failed — see the `added` /
|
||||
`failed` tallies + per-item `results[].status` for partial outcomes.)
|
||||
- **Structured errors.** Failures arrive as `CODE: message (retriable=…)`, where `CODE` is one of
|
||||
`AUTH`, `RATE_LIMITED`, `NOT_FOUND`, `VALIDATION`, `TIMEOUT`, `NETWORK`, `SERVER`, `RPC`,
|
||||
`CONFIG`, `NOTEBOOK_LIMIT`, `ARTIFACT_TIMEOUT`, `SOURCE_MUTATION`, `ERROR`, or `UNEXPECTED`. The
|
||||
`retriable` flag tells an agent whether a retry could succeed (e.g. `RATE_LIMITED`, `TIMEOUT`,
|
||||
`NETWORK`). Many errors also carry an actionable `hint` (e.g. `AUTH → run notebooklm login`); a
|
||||
near-miss name lookup puts its `Did you mean: …` candidates (title + id) in that hint (see
|
||||
**Name *or* ID** above).
|
||||
|
||||
## Workflows
|
||||
|
||||
The examples below are MCP **tool calls** an agent makes (not shell commands).
|
||||
|
||||
### Add sources and ask a question
|
||||
|
||||
```text
|
||||
nb = notebook_create(title="Quantum Computing")
|
||||
source_add(notebook="Quantum Computing", source_type="url", url="https://arxiv.org/abs/...")
|
||||
source_add(notebook="Quantum Computing", source_type="text", title="Notes", text="...")
|
||||
source_wait(notebook="Quantum Computing") # block until sources finish processing
|
||||
chat_ask(notebook="Quantum Computing", question="What are the open problems?")
|
||||
```
|
||||
|
||||
`source_wait` returns a structured aggregate: the four buckets (`ready` carries
|
||||
`source_view` rows; `timed_out`/`failed`/`not_found` carry `{source_id, error}`)
|
||||
plus explicit `*_count` scalars and a `total_count` for at-a-glance triage — so a
|
||||
client reads the counts without folding `len()` over every array. The counts are
|
||||
additive; the arrays stay for backward compatibility. `ok` is `true` iff every
|
||||
error bucket is empty, and `total_count` = `ready_count` + `timed_out_count` +
|
||||
`failed_count` + `not_found_count`:
|
||||
|
||||
```text
|
||||
source_wait(notebook="Quantum Computing")
|
||||
# → {"notebook_id": ..., "ok": false,
|
||||
# "ready": [{"id": ..., "title": "Notes", "kind": "pasted_text", "status_label": "ready"}],
|
||||
# "timed_out": [{"source_id": ..., "error": "..."}],
|
||||
# "failed": [],
|
||||
# "not_found": [],
|
||||
# "ready_count": 1, "timed_out_count": 1, "failed_count": 0,
|
||||
# "not_found_count": 0, "total_count": 2}
|
||||
```
|
||||
|
||||
`source_type` is one of `url`, `text`, `file` (local `path`), `drive` (a
|
||||
`document_id` + a **required** `mime_type`, one of
|
||||
`google-doc`/`google-slides`/`google-sheets`/`pdf` — there is no default, since
|
||||
defaulting a non-Doc Drive file to `google-doc` fails the import), or `youtube`.
|
||||
URL and YouTube adds reject
|
||||
internal/loopback hosts by default; pass `allow_internal=true` only for
|
||||
deliberate local NotebookLM tests. `chat_ask` continues the most-recent
|
||||
conversation unless you pass a `conversation_id`.
|
||||
|
||||
To add ONE source and block until it finishes processing in a single call, use
|
||||
`source_add_and_wait` — it composes single-mode `source_add` + `source_wait`, so
|
||||
you skip the add→wait round-trip. It takes the same single-mode add inputs plus
|
||||
the `timeout`/`interval` wait knobs, and returns the `source_wait` aggregate plus
|
||||
a top-level `source_id` (always present — the source persists even if the wait
|
||||
times out or the import fails, so you can retry or delete it):
|
||||
|
||||
```text
|
||||
source_add_and_wait(notebook="Quantum Computing", source_type="url",
|
||||
url="https://arxiv.org/abs/...")
|
||||
# → {"notebook_id": ..., "ok": true,
|
||||
# "ready": [{"id": ..., "title": "...", "kind": "web_page", "status_label": "ready"}],
|
||||
# "timed_out": [], "failed": [], "not_found": [],
|
||||
# "ready_count": 1, "timed_out_count": 0, "failed_count": 0,
|
||||
# "not_found_count": 0, "total_count": 1, "source_id": ...}
|
||||
```
|
||||
|
||||
It is single-source only (no `urls` batch) and cannot one-shot a **remote** `file`
|
||||
upload (that upload is a separate step — use `source_add(source_type="file")` then
|
||||
`source_wait`, or `source_upload_bytes` for a tiny file).
|
||||
|
||||
To ingest many URLs at once, pass `urls` (batch mode) instead of `source_type`
|
||||
— one call instead of one round-trip each. The response is an explicit per-item
|
||||
list so a partial failure is never hidden behind a single success flag:
|
||||
|
||||
```text
|
||||
source_add(notebook="Quantum Computing", urls=[
|
||||
"https://arxiv.org/abs/2401.00001",
|
||||
"https://www.youtube.com/watch?v=...",
|
||||
])
|
||||
# → {"notebook_id": ..., "added": 2, "failed": 0,
|
||||
# "results": [{"input": "https://arxiv.org/abs/2401.00001", "status": "added", "source_id": ..., "title": ...},
|
||||
# {"input": "https://www.youtube.com/watch?v=...", "status": "added", "source_id": ..., "title": ...}]}
|
||||
```
|
||||
|
||||
Batch mode is URL-only (a non-URL entry is reported as a per-item `VALIDATION`
|
||||
error, never added as text); `source_type`/`url`/`text`/`title`/`path`/
|
||||
`document_id`/`mime_type` are not valid with `urls`, but `allow_internal`
|
||||
applies to every entry.
|
||||
|
||||
### Content-sanity warnings on ready web pages
|
||||
|
||||
A dead link, [soft-404](https://en.wikipedia.org/wiki/HTTP_404#Soft_404), or
|
||||
paywalled page frequently ingests as a **READY** source with little-to-no
|
||||
extractable text — a "ghost source" that add-time status can't catch because a
|
||||
soft-404 serves HTTP 200. `source_wait` — and batch `source_add(urls=[...])` for
|
||||
an item that is *already* READY the moment it returns (single-mode `source_add`
|
||||
adds asynchronously, so it never runs this check) — attaches a non-blocking,
|
||||
advisory `warning` to such a source. The check is **best-effort and never
|
||||
rejects**: the source stays READY, `ok` stays `true`, and any fetch failure
|
||||
(including a >5s slow `source_read`) degrades to no warning rather than breaking
|
||||
the wait.
|
||||
|
||||
It fires on a **web-page source only** (`kind == "web_page"`) via two body-only
|
||||
signals — the title is never scanned:
|
||||
|
||||
| Signal | Threshold | Warning contains |
|
||||
|--------|-----------|------------------|
|
||||
| **char-thin** | indexed text shorter than **100 characters** | `"little/no text extracted (N chars) …"` |
|
||||
| **dead-link boilerplate** | **indexed text** shorter than **2000 characters** that (casefolded) contains any of the phrases below | `"ingested as ready (N chars) but the body matches a dead-link / error-page pattern …"` |
|
||||
|
||||
The full dead-link phrase set (the complete list, so you can build a fixture that
|
||||
trips it): `broken link`, `page not found`, `page isn't available`, `page does
|
||||
not exist`, `page no longer available`, `no longer available`, `error 404`, `404
|
||||
not found`, `whoops!`.
|
||||
|
||||
Both gates measure the source's **indexed text** length (`char_count` from a
|
||||
`source_read` with `detail="full"`), not the raw HTTP response — a large HTML
|
||||
page that indexes to little text is still caught. The 2000-char gate is what
|
||||
keeps the weaker phrases safe: a page whose indexed text is 2000 chars or longer
|
||||
is never phrase-scanned (so `broken link` in a real article about broken links,
|
||||
or a shop's `no longer available`, does not false-positive), and the phrases are
|
||||
all multi-word / anchored — no bare `404` or `not found`. Every warning ends with
|
||||
`verify with source_read (detail="full").` (trailing period included).
|
||||
|
||||
**To exercise the warning branch** (the reason this is documented): note that a
|
||||
`text` source — even an empty one — is *never* flagged; only a `web_page` under
|
||||
the thresholds above is. So the reliable trigger is a URL that resolves to a
|
||||
near-empty or soft-404 page. To unit-test your own handling of the branch
|
||||
without a live URL, mock the source's fetched body under the threshold and assert
|
||||
the warning shape — copy the pattern from
|
||||
[`tests/unit/mcp/test_sources.py`](../tests/unit/mcp/test_sources.py) (see
|
||||
`test_source_wait_thin_web_page_warns`, `test_source_wait_soft_404_body_phrase_warns`,
|
||||
and the `_THIN_SOURCE_CHAR_THRESHOLD` boundary test).
|
||||
|
||||
### Generate and download a studio artifact
|
||||
|
||||
```text
|
||||
task = studio_generate(notebook="Quantum Computing", artifact_type="audio")
|
||||
studio_status(notebook="Quantum Computing", task_id="<task_id from above>") # poll until complete
|
||||
studio_download(notebook="Quantum Computing", artifact_type="audio", path="podcast.mp3")
|
||||
|
||||
# Target a specific/older artifact instead of the latest-by-type (full ID or unique prefix):
|
||||
studio_download(notebook="Quantum Computing", artifact_type="audio", path="old_podcast.mp3", artifact_id="aaaaaaaa-aaaa")
|
||||
|
||||
# Per-kind styling options are agent-settable, e.g. a custom-styled video:
|
||||
studio_generate(notebook="Quantum Computing", artifact_type="video",
|
||||
style="custom", style_prompt="hand-drawn diagrams")
|
||||
```
|
||||
|
||||
`studio_download`'s `output_format` overrides the download file format, but only
|
||||
some artifact types have a format axis:
|
||||
|
||||
| `artifact_type` | Supported `output_format` |
|
||||
| --- | --- |
|
||||
| `slide-deck` | `pdf` (default), `pptx` |
|
||||
| `quiz`, `flashcards` | `json` (default), `markdown`, `html` |
|
||||
| `audio`, `video`, `infographic`, `report`, `mind-map`, `data-table` | none — omit `output_format` |
|
||||
|
||||
Passing `output_format` to a type with no format axis (e.g. `report` +
|
||||
`markdown`) is a validation error that says `supported formats: default only`
|
||||
rather than silently ignoring it, and the message is identical whether the
|
||||
download runs over stdio (`path`) or the remote signed-URL connector.
|
||||
|
||||
`artifact_type` is one of `audio`, `video`, `cinematic-video`, `slide-deck`, `quiz`, `flashcards`,
|
||||
`infographic`, `data-table`, `mind-map`, `report`. Each kind's styling options are agent-settable
|
||||
(matching the CLI flags): `audio_format` / `audio_length` (audio); `video_format` / `style` /
|
||||
`style_prompt` (video — `style` / `style_prompt` are rejected for `video_format` `cinematic` and
|
||||
`short`, which use a fixed visual style); `deck_format` / `deck_length` (slide-deck); `quantity` / `difficulty`
|
||||
(quiz, flashcards); `orientation` / `detail` / `style` (infographic); `map_kind` (mind-map);
|
||||
and `report_format` (report). `cinematic-video` and `data-table` take no per-kind options. An
|
||||
option is valid only for its own kind — passing one to a different `artifact_type` is a
|
||||
validation error, not a silent no-op.
|
||||
|
||||
### Run deep research and import the findings
|
||||
|
||||
```text
|
||||
task = research_start(notebook="Quantum Computing", query="post-quantum cryptography", source="web", mode="deep")
|
||||
research_status(notebook="Quantum Computing", poll_task_id=task["poll_task_id"])
|
||||
research_import(notebook="Quantum Computing", poll_task_id=task["poll_task_id"])
|
||||
```
|
||||
|
||||
`source` is `web` or `drive`; `mode` is `fast` or `deep`. Pass the
|
||||
`poll_task_id` returned by `research_start` — under the **same** parameter name,
|
||||
`poll_task_id` — when polling, importing, or cancelling, so the value copies
|
||||
verbatim from one tool's output into the next and the request is pinned to the
|
||||
intended research task (for a **deep** run it is the `report_id`; the raw
|
||||
`task_id` is an unpollable sessionId). Omitting the pin on `research_status` is
|
||||
allowed only when the notebook has a single in-flight task. `research_status`
|
||||
omits the large report by default — pass `include_report=true` to fetch it once
|
||||
`completed`.
|
||||
|
||||
> **Deprecated (removed in v0.9.0):** `research_status`/`research_import` also
|
||||
> accept the old `task_id` name and `research_cancel` the old `run_id` name as
|
||||
> aliases for `poll_task_id`. Passing an alias still works but emits a
|
||||
> `DeprecationWarning` and adds a `deprecation` note to the result — switch to
|
||||
> `poll_task_id`. See [docs/deprecations.md](deprecations.md).
|
||||
|
||||
## Tool reference
|
||||
|
||||
| Domain | Tools |
|
||||
|--------|-------|
|
||||
| **Notebooks** | `notebook_list(limit?, offset?)` · `notebook_create(title)` · `notebook_describe(notebook, include_metadata?)` (AI description; `include_metadata=true` adds a `metadata` block with notebook details + source list) · `notebook_rename(notebook, new_title)` · `notebook_delete(notebook, confirm)` |
|
||||
| **Sources** | `source_list(notebook, status?, label?, detail?, limit?, offset?)` (each source has string `kind`/`status_label`; `status` filters to one of ready\|processing\|error\|preparing — e.g. `status="error"` finds failed imports; `detail=compact` returns a low-token roster of just `id`/`title`/`kind`/`status_label`/`created_at`) · `source_read(notebook, source, detail?, output_format?, max_chars?, offset?)` (`detail=full` (default) → metadata + a bounded slice of the indexed text: `max_chars` caps `content` (default 10k), `offset` pages, plus a `truncated` flag and the full `char_count`; `detail=summary` → low-token triage: AI summary **+ keywords**, not the body; `output_format`: text\|markdown) · `source_rename(notebook, source, new_title)` · `source_delete(notebook, source, confirm)` · `source_wait(notebook, source?, timeout, interval)` (a READY web page with thin/empty text, or a short body matching a dead-link / soft-404 boilerplate pattern, carries a non-blocking `warning`) · `source_add(notebook, source_type, ..., allow_internal?)` (single; echoes `kind`/`status_label`, flags a failed import inline with a `warning`) / `source_add(notebook, urls=[...], allow_internal?)` (batch → per-item `results`; a synchronously-ready web-page item may also carry the same content-sanity `warning`) · `source_add_and_wait(notebook, source_type, ..., timeout?, interval?)` (single-mode `source_add` + `source_wait` in one call → the `source_wait` aggregate plus a top-level `source_id`; not for batch or a remote `file` upload) |
|
||||
| **Chat** | `chat_ask(notebook, question?, conversation_id?, references?, source_ids?, history?, suggest_followups?)` (`references`: lite\|full; never returns the raw debug blob; `source_ids` scopes to specific sources — list, JSON-array string, or comma string; omit for all; `history`>0 also returns up to N prior `{question, answer}` pairs — omit `question` to recall only; `suggest_followups=true` also returns `suggested_prompts` (3 questions to ask — works question-less too)) · `chat_configure(notebook, chat_mode?, goal?, response_length?)` (`chat_mode`: default\|learning-guide\|concise\|detailed — a preset, mutually exclusive with `goal`/`response_length`; a **partial** custom call sets just `goal` or just `response_length` and **merges** with the current settings — the omitted field is preserved, not reset; only a bare call, no preset and neither field, is rejected) · `suggest_prompts(notebook, surface?, source_ids?, query?)` (READ_ONLY; `surface`: ask\|audio-deep-dive\|audio-brief\|audio-critique\|audio-debate\|video-explainer\|video-short\|quiz\|flashcards — returns `{title, prompt}` suggestions to steer that studio surface; `ask` (default) = chat questions) |
|
||||
| **Notes** | `note_save(notebook, note?, title?, content?)` (upsert: omit `note` to **create** — `title` AND `content` required; pass a `note` ref to **update** — `title` and/or `content`, title-only = rename). Reading and deleting notes fold into the Studio row below. |
|
||||
| **Studio** | `studio_list(notebook, item?, kind?, detail?, limit?, offset?)` (the unified Studio panel — **notes AND artifacts** merged into one `items` list; each item has `id`/`title`/`type` where `type` is `note` or a hyphenated artifact kind; artifacts add `status_label`/`url`; `detail=summary` (default) gives each note a bounded `content_preview` + full-body `char_count` to keep a discovery listing low-token, `detail=full` returns the whole note `content`, `detail=compact` collapses every item to `id`/`title`/`type`/`status_label`/`created_at`; `kind` filters to one `type`; `item` fetches one note-or-artifact by ref as a 1-element list, always with the note's full `content`) · `studio_generate(notebook, artifact_type, …)` · `studio_status(notebook, task_id)` · `studio_get_prompt(notebook, artifact)` (the free-text prompt an artifact was generated from; `null` if none) · `studio_download(notebook, artifact? \| artifact_type?, path?, output_format?, artifact_id?)` (target by `artifact` name-or-id ref **or** by `artifact_type` [+ `artifact_id` for a specific one, else latest]) · `studio_rename(notebook, item, new_title)` (cross-type: renames a note OR an artifact resolved from the merged list) · `studio_retry(notebook, artifact)` (re-run a failed artifact in place; task_id == artifact_id) · `studio_delete(notebook, item, confirm)` (cross-type: deletes a note OR an artifact resolved from the merged list) |
|
||||
| **Research** | `research_start(notebook, query, source, mode)` (returns `poll_task_id` — the one id status/import/cancel drive off) · `research_status(notebook, poll_task_id?, include_report?, report_max_chars?, source_limit?, source_offset?)` (report + per-source `report_markdown` omitted unless `include_report`) · `research_import(notebook, poll_task_id)` · `research_cancel(notebook, poll_task_id)` (sends the cancel unless the run is already terminal → `cancel_requested`). The old `task_id` / `run_id` param names are deprecated aliases for `poll_task_id`, removed in v0.9.0 |
|
||||
| **Sharing** | `share_status(notebook)` (is_public/access/share_url/shared_users; enums as string labels; `view_level` omitted — the read API can't report it) · `share_set_access(notebook, public?, view_level?, confirm)` (link settings; `view_level`: full\|chat, echoed back only when set; `confirm` gates public widening restricted→public) · `share_set_user(notebook, email, permission?, notify?, message?, confirm)` (upsert grant; `permission`: editor\|viewer; `notify` defaults `false`; `confirm` gates every grant) · `share_remove_user(notebook, email, confirm)` |
|
||||
| **Server** | `server_info(include_account?)` — version + local auth health; `include_account=true` adds an `account` block: signed-in identity (`email`, `authuser`) plus notebook/source limits, the subscription `tier` (opaque enum, e.g. `1`=Free/`2`=Pro, `null` on legacy responses; per-tier limits in [quota-limits.md](quota-limits.md)), and global `output_language` for quota pacing + language context (best-effort; identity is network-free from the profile, the quota fields need a live session). `email` is real account PII, returned only under this opt-in flag |
|
||||
|
||||
Tools that only read are annotated read-only; the destructive tools (the three `*_delete` tools plus `share_remove_user`) are annotated destructive
|
||||
and require `confirm`. A host that honors MCP annotations can auto-allow the read-only calls and
|
||||
gate the destructive ones.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **`AUTH` errors / "not authenticated".** Run `notebooklm login` (or `notebooklm -p <profile> login`)
|
||||
in a terminal, then restart the server. Check with the `server_info` tool, which reports auth health.
|
||||
- **`uvx` / `uv` not found.** Install uv: `curl -LsSf https://astral.sh/uv/install.sh | sh` (macOS/Linux)
|
||||
or `powershell -c "irm https://astral.sh/uv/install.ps1 | iex"` (Windows). The desktop launcher also
|
||||
searches common install dirs beyond `PATH`.
|
||||
- **Client doesn't see the tools.** Confirm the config was written (`notebooklm mcp install <client>`)
|
||||
and **restart the client** — most hosts only read MCP config at startup.
|
||||
- **Wrong account.** The server binds one profile per process. Start it with `--profile <name>`, or set
|
||||
`NOTEBOOKLM_PROFILE`. See [configuration.md](configuration.md#multiple-accounts).
|
||||
- **`RATE_LIMITED`.** NotebookLM enforces per-account quotas; the error is `retriable=true` — back off
|
||||
and retry.
|
||||
|
||||
## See also
|
||||
|
||||
- [installation.md](installation.md#running-the-mcp-server-mcp-extra) — the `mcp` extra + run/connect summary
|
||||
- [`desktop-extension/README.md`](../desktop-extension/README.md) — one-click Claude Desktop `.mcpb` bundle (prebuilt, attached to each stable release)
|
||||
- [configuration.md](configuration.md) — profiles, multi-account, storage
|
||||
- [cli-reference.md](cli-reference.md) — the equivalent CLI commands
|
||||
@@ -0,0 +1,61 @@
|
||||
# `otmP3b` (GeneratePromptSuggestions) mode → surface map
|
||||
|
||||
Evidence record for the `suggest_prompts` mode enum (field-4 `C0` of the
|
||||
`SUGGEST_PROMPTS` / `otmP3b` request). Backs the MCP `suggest_prompts` tool's
|
||||
`surface` labels and the client cap (`_notebook_payloads._PROMPT_SUGGESTIONS_MODE_MAX = 10`).
|
||||
Investigated live 2026-07-01 (#1726); supersedes the earlier output-only #1612 guess.
|
||||
|
||||
## Map
|
||||
|
||||
| mode | surface · format | how established |
|
||||
|---|---|---|
|
||||
| 1 | Audio · Deep Dive | **browser-verified** — opened the Audio Overview *Customize* dialog headless (Playwright + stored `storage_state` auth), clicked the "Deep Dive" format card, decoded the `otmP3b` request's trailing mode |
|
||||
| 2 | Audio · Brief | browser-verified (Brief card) |
|
||||
| 3 | Video · Explainer | real web capture — `otmP3b` request while customizing a Video Explainer |
|
||||
| 4 | Chat · ask (default) | client live-probe — returns "ask about the content" chat questions; the web chat default |
|
||||
| 5 | Audio · Critique | browser-verified (Critique card) |
|
||||
| 6 | Audio · Debate | browser-verified (Debate card); output = debate scaffolding |
|
||||
| 7 | — unidentified | no UI surface sends it; excluded from the tool |
|
||||
| 8 | Quiz | client live-probe — returns quiz-generation prompts |
|
||||
| 9 | Flashcards | client live-probe — returns flashcard prompts |
|
||||
| 10 | Video · Short | real web capture — controlled pair (same notebook + source, mode 3 vs 10) |
|
||||
| 0, 11+ | INTERNAL (server error) | out of range |
|
||||
|
||||
## Two axes (why the labels are honest)
|
||||
|
||||
There are two *different* measurements, and they are **consistent**, not contradictory:
|
||||
|
||||
- **Surface axis (#1726):** which mode a studio *Customize* dialog SENDS for its
|
||||
format's suggestions. This is request-side wiring — the `otmP3b` mode the UI
|
||||
emits per format. It's how the labels are assigned (a card labeled "Deep Dive"
|
||||
sends mode 1, so mode 1 = the Audio Deep Dive suggestion surface).
|
||||
- **Output axis (#1612, 2026-06-20 live A/B):** what the backend RETURNS per mode.
|
||||
Modes 5/6/8/9 (critique / debate / quiz / flashcards) are format-distinctive in
|
||||
the returned text; modes 1/2/3/10 return content-direction prompts
|
||||
(persona / format / topic) that read ~like the mode-4 default.
|
||||
|
||||
Reconciliation: for deep-dive / brief / explainer / short, NotebookLM steers the
|
||||
format via **content direction**, not format jargon — so those surfaces' suggestion
|
||||
*text* is general by design, even though the *surface* (which mode the format's
|
||||
dialog sends) is distinct and verified. The tool's `surface` labels therefore name
|
||||
the **surface** (the format whose dialog sends the mode), not a promise about the
|
||||
returned tone. Modes 5/6/8/9 also differ in output; 1/2/3/10 are the "same shape,
|
||||
different surface" cases.
|
||||
|
||||
## Not exposed / not supported
|
||||
- Mode **7**: no UI surface sends it; unmapped.
|
||||
- **data-table / infographic / mind map / slide deck**: their Customize dialogs
|
||||
exist but fire NO `otmP3b` (no suggestion surface). Modes 8/9 (quiz/flashcards)
|
||||
are backend-reachable via this RPC but the web UI doesn't wire their dialogs to
|
||||
it — still valid to expose since the RPC serves them.
|
||||
- **Report** suggestions: a *separate* RPC (`GET_SUGGESTED_REPORTS` / `ciyUvf`),
|
||||
not `otmP3b`.
|
||||
|
||||
## Method (reproducible)
|
||||
Headless Playwright loaded the authenticated app via the repo's `storage_state`
|
||||
auth (`browser.new_context(storage_state=…)`), navigated to a notebook, and clicked
|
||||
each artifact's `button[aria-label="Customize {Type}"]` chevron (NOT the card body —
|
||||
that generates). For Audio, each format card inside the dialog was clicked while
|
||||
capturing the `otmP3b` request post-data (`f.req` → trailing mode int). Video modes
|
||||
came from real browser network captures (`otmP3b` requests). Chat/quiz/flashcards
|
||||
were client live-probes of `client.notebooks.suggest_prompts(mode=…)`.
|
||||
@@ -0,0 +1,140 @@
|
||||
# REST server vs MCP server — gap review (2026-07-02)
|
||||
|
||||
**Inputs (4 lenses):** 2 native Claude (capability-parity matrix · shared-capability contract
|
||||
divergence) + Codex CLI (REST security/hardening vs MCP) + agy CLI (unique-to-each inventory). All
|
||||
read the current `src/notebooklm/{server,mcp}/` tree. Both adapters are thin wrappers over the same
|
||||
`src/notebooklm/_app/` cores, so a "gap" = an `_app/` capability one adapter exposes and the other
|
||||
doesn't (or exposes with a materially different contract).
|
||||
|
||||
## Headline
|
||||
|
||||
The gap is **overwhelmingly one-directional**. The REST server is missing ~11 capability areas the
|
||||
MCP server has; the MCP server is missing **~zero** real capabilities REST has. On top of that, REST
|
||||
lags MCP on a whole tier of **hardening** (confirm gates, redaction, resource caps, poll semantics).
|
||||
|
||||
- **MCP → REST (things REST lacks):** deep research (entire core), source **content** read,
|
||||
`chat_configure`, studio lifecycle (delete/retry/rename/get_prompt), `source_wait`,
|
||||
`suggest_prompts`, notebook/source rename, `server_info`, Drive/YouTube/batch source-add.
|
||||
- **REST → MCP (things MCP lacks):** **none of substance.** REST's dedicated `/notes` CRUD looks
|
||||
unique but MCP covers all of it through the cross-type Studio surface (`note_save` upsert +
|
||||
`studio_list` read/enumerate with `kind="note"` + `studio_delete`). This is an interface-shape
|
||||
difference, not a capability gap. *(Corrects one lens that called MCP a "write-only notes client" —
|
||||
`_studio_items.py:95-111` reads `client.notes.list`; `studio_delete` deletes notes.)*
|
||||
|
||||
**Bottom line:** REST is **not** a viable substitute for MCP today. It can create notebooks, add
|
||||
url/text/file sources, run a blocking chat, and generate + download artifacts — but a REST client
|
||||
**cannot read what a source actually says, cannot run deep research, and cannot tune chat.** It is
|
||||
also the softer target of the two on security.
|
||||
|
||||
---
|
||||
|
||||
## A. Capability gaps — MCP has it, REST doesn't
|
||||
|
||||
Ranked by user impact. Backing `_app/` core in parens.
|
||||
|
||||
| # | Gap (REST-missing) | `_app/` core | Impact | Close cost |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **Deep research** (start/status/cancel/import) — no research router mounted at all | `research`, `source_research` | **HIGH** | New `/research` router |
|
||||
| 2 | **Source content read** — `GET /sources/{id}` returns metadata/status only (`get_or_none`), never the extracted fulltext MCP's `source_read` serves | `source_content` | **HIGH** | New content route |
|
||||
| 3 | **`chat_configure`** — REST chat is locked to notebook defaults (no goal/persona/length) | `chat.execute_configure` | **MED-HIGH** | New route |
|
||||
| 4 | **Studio delete** — REST generates artifacts but can never remove them | `artifacts.delete_artifact` | **MED** | New route |
|
||||
| 5 | **Studio retry** — a failed generation can't be retried in place | `generate_retry` | **MED** | New route |
|
||||
| 6 | **`source_wait`** — no readiness primitive; REST callers hand-roll polling | `source_wait` | **MED** | New route |
|
||||
| 7 | **Drive/YouTube/batch source-add** — REST has only url/text/file, one at a time | `source_add` | **MED** | Extend routes/models |
|
||||
| 8 | **`suggest_prompts`** — no recommended-question affordance | (client namespace) | **LOW-MED** | New route |
|
||||
| 9 | **`studio_get_prompt`** — can't retrieve the prompt an artifact was generated with | `artifacts.get_artifact_prompt` | **LOW** | New route |
|
||||
| 10 | **notebook/source/artifact rename** — no PATCH/rename verbs | `notebooks`, `source_mutations`, `artifacts` | **LOW** | Trivial passthrough routes |
|
||||
| 11 | **`server_info`** — REST has only unauthed `/healthz` liveness, no account/version block | `profile`, `auth_check` | **LOW** | New route |
|
||||
|
||||
## B. Capability gaps — REST has it, MCP doesn't
|
||||
|
||||
**Effectively none.** REST's `/notes` list/get/put/delete map to MCP's `note_save` + `studio_list`
|
||||
(`kind="note"`, single-fetch by ref returns `content`) + `studio_delete`. Same underlying
|
||||
`_app.notes` ops, different URL shape. Not a gap.
|
||||
|
||||
---
|
||||
|
||||
## C. Contract divergence on shared capabilities (both surfaces expose these, but differently)
|
||||
|
||||
Consensus items (2+ lenses) first.
|
||||
|
||||
1. **Async generation poll semantics are genuinely incompatible — HIGH.** MCP `studio_status` is
|
||||
**stateless**: any `task_id` polls its real status and terminal states (FAILED/REMOVED) come back
|
||||
as fields. REST records the id in a **process-local in-memory `PendingRegistry`**
|
||||
(`server/_pending.py:18-22`) and maps unknown→404, REMOVED→410, FAILED→409. A real, still-running
|
||||
task the process didn't create (or created before a restart) returns **404 on REST but live status
|
||||
on MCP**. (`artifacts.py:298-324`) *(Lens B)*
|
||||
|
||||
2. **Destructive + outward ops: MCP confirm-gated, REST fires immediately — HIGH.** *(Codex + Lens B
|
||||
consensus)* MCP two-steps every delete and gates public-widening / user-grant behind `confirm=`.
|
||||
REST: `POST /share/public` makes a notebook public, and `POST /share/users` grants access **and
|
||||
emails the user**, each on a single unconfirmed request. Deletes are bare `DELETE` verbs (defensible
|
||||
HTTP idiom) but the **outward sharing** ops have no guard at all. (`share.py:74,86,115`)
|
||||
|
||||
3. **`notify` default is inverted — MED.** *(Codex + Lens B consensus)* `share_set_user` defaults
|
||||
`notify=False` on MCP; REST `POST /share/users` defaults `notify=True` (`share.py:31`) — the same
|
||||
grant emails a third party by default on REST, silently on MCP. (REST's `PATCH /users/{email}`
|
||||
carries no notify flag at all.)
|
||||
|
||||
4. **Output leaks MCP intentionally strips — MED.** *(Lens B)* REST `to_jsonable`s raw results:
|
||||
sharing status ships bare integer enums (`access=1`, `permission=3`) instead of MCP's string labels
|
||||
(`share.py:71,83,143`); chat ships the `raw_response` debug blob MCP pops (`chat.py:47`). Source
|
||||
list ships raw integer status/type codes with no `kind`/`status_label`.
|
||||
|
||||
5. **Error contract thinner + a redaction hole — MED.** *(Codex + Lens B)* MCP errors carry
|
||||
`retriable` + `hint` and redact home paths / file tokens; REST returns `{category, message}` + HTTP
|
||||
status with **no `retriable`, no `hint`, and no home-path redaction** (`server/_errors.py:94-107`).
|
||||
A file-upload path error can leak `/home/<user>/…` in a REST body.
|
||||
|
||||
6. **Identifiers: MCP resolves name-or-id-prefix, REST is id-only — MED.** *(Lens B)* REST resolvers
|
||||
are literal passthroughs (`_passthrough.py:27-31`); a REST caller must hold canonical UUIDs.
|
||||
|
||||
7. **Lists: MCP paginated (limit/offset/total/has_more), REST returns the whole collection raw — MED.**
|
||||
*(Lens B)* Large notebooks/accounts dump unbounded raw-code payloads over REST.
|
||||
|
||||
8. **Note create/update ergonomics inverted — MED.** *(Lens B)* MCP requires title+content on create,
|
||||
allows partial update; REST allows an empty-body create (`title="New Note"`, `content=""`) but
|
||||
forces a full title+content replacement on `PUT` (`notes.py:35-47,70-79`).
|
||||
|
||||
> Verified non-issue: the `source_ids` **None=all vs []=none** footgun is honored correctly on **both**
|
||||
> sides (`artifacts.py:269` + `_passthrough.py:52` mirror `studio.py:424` + `173`). No divergence.
|
||||
|
||||
---
|
||||
|
||||
## D. Security / hardening asymmetry (REST lags MCP)
|
||||
|
||||
From Codex, all REST-side, all citing an MCP protection REST doesn't replicate:
|
||||
|
||||
| Sev | Finding | Where |
|
||||
|---|---|---|
|
||||
| HIGH | Public-share + user-grant widen access with no confirm gate (dup of C.2) | `share.py:74,86,115` |
|
||||
| HIGH | Share invite **emails by default** (`notify=True`) (dup of C.3) | `share.py:31` |
|
||||
| HIGH | Destructive deletes have no confirm/preview | `notebooks.py:62`, `sources.py:235`, `notes.py:82` |
|
||||
| MED | Multipart uploads spool to disk before the size cap runs (chunked/no-`Content-Length` bypass) | `app.py:97`, `sources.py:204` |
|
||||
| MED | Artifact downloads unbounded + cleanup only via background task — no disconnect-safe slot cap like MCP | `artifacts.py:339,353,382` |
|
||||
| MED | `--token` CLI arg writes the bearer token where `ps` can read it; MCP is env-only on purpose | `__main__.py:107,152` |
|
||||
| MED | Loopback guard is launcher-only + spoofable `Host` header in-app; running the ASGI app directly on `0.0.0.0` is reachable | `__main__.py:55,157`, `_auth.py:114` |
|
||||
| LOW | Error redaction lacks MCP's home-path / file-link patterns (dup of C.5) | `_errors.py:94` |
|
||||
| LOW | Artifact generate silently accepts wrong-kind options and drops mind-map `instructions` | `artifacts.py:264,275` |
|
||||
| LOW | Upload filename sanitization weaker than MCP's `_safe_upload_name` | `sources.py:57` |
|
||||
|
||||
*(Auth token comparison itself: worth confirming `server/_auth.py` uses `compare_digest` — Codex
|
||||
didn't flag a timing leak, so assume present; verify if hardening this.)*
|
||||
|
||||
---
|
||||
|
||||
## Recommended priority if REST is to be taken seriously
|
||||
|
||||
**Tier 1 (makes REST usable at all):** source content read (A.2), deep research router (A.1),
|
||||
`chat_configure` (A.3). Without the first, a REST client is blind to its own documents.
|
||||
|
||||
**Tier 2 (safety parity):** confirm-gate outward sharing + flip `notify` default (C.2/C.3/D),
|
||||
port MCP's home-path redaction (C.5/D), add download concurrency cap (D). These are small diffs that
|
||||
close the sharpest security asymmetries.
|
||||
|
||||
**Tier 3 (completeness):** studio delete/retry (A.4/A.5), `source_wait` (A.6), rename verbs (A.10),
|
||||
pagination (C.7), fix the stateful-poll 404 (C.1 — either document it or back the registry with a real
|
||||
status probe on cache-miss).
|
||||
|
||||
**Cheapest high-value single fix:** flip `notify` default to `False` and confirm-gate `POST
|
||||
/share/public` — one-line-ish changes that remove two HIGH outward-facing footguns.
|
||||
+2770
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,200 @@
|
||||
# NotebookLM quota & tier limits (reference)
|
||||
|
||||
A human-reference table of NotebookLM's **static, published plan limits** per tier — how many
|
||||
notebooks, sources, chats, and studio artifacts each subscription level allows.
|
||||
|
||||
**What this is NOT:** live per-account *remaining* counts or *reset* timestamps. NotebookLM does
|
||||
**not** expose those through any known RPC — `GET_USER_SETTINGS` carries only the static plan
|
||||
limits, not usage counters. See [#1825](https://github.com/teng-lin/notebooklm-py/issues/1825) for
|
||||
the research trail. This document is therefore **prose reference, not shipped code** (see
|
||||
[Why this lives in docs](#why-this-lives-in-docs-not-code)).
|
||||
|
||||
> ⚠️ **Captured from Google's public pages on the dates noted below. Every table Google publishes is
|
||||
> headed "Usage Limits (Subject to Change)".** Google restructured consumer tiers as recently as
|
||||
> May 2026 and has changed *enforced* limits without updating the published tables (see
|
||||
> [Caveats](#caveats)). Treat these numbers as a dated snapshot, re-verify against the source links
|
||||
> before relying on a specific value, and prefer the authoritative live signal —
|
||||
> [`AccountLimits.tier`](#relationship-to-accountlimitstier) — over hard-coding any of this.
|
||||
|
||||
## Relationship to `AccountLimits.tier`
|
||||
|
||||
`(await client.settings.get_account_limits()).tier` (and the `tier` field in MCP/REST
|
||||
`server_info(include_account=True)`) returns the subscription tier as an **opaque integer** read
|
||||
from `GET_USER_SETTINGS` `limits[4]`. This table is what those integers *mean* in quota terms.
|
||||
|
||||
| `tier` int | Plan | Live-confirmed? |
|
||||
|---|---|---|
|
||||
| `1` | Standard / Free | ✅ yes (source limit 50 matches Google) |
|
||||
| `2` | Pro | ✅ yes (source limit 300 matches Google) |
|
||||
| `4` | Plus | decoded from the web pro-badge bundle |
|
||||
| `3` | Ultra (20 TB) | decoded |
|
||||
| `6` | Ultra (30 TB) | decoded |
|
||||
| `5` | "Expanded" | decoded — aligns with the **Workspace "Expanded"** access level below (not a consumer plan, which is why it is absent from Google's consumer page) |
|
||||
|
||||
The `tier` int is an **opaque key, not an ordinal** — `4` (Plus) is numerically higher than `2`
|
||||
(Pro) but a lower plan. Look it up in this table; never compare tier ints with `<`/`>`.
|
||||
|
||||
**Two things the tier int alone cannot tell you:**
|
||||
|
||||
1. **Consumer vs Workspace vs Enterprise.** Pro (consumer), Higher (Workspace), and Enterprise all
|
||||
report `notebook_limit=500` / `source_limit=300`, so the two quota numbers do not disambiguate
|
||||
the surface. The three tables below diverge — pick the row that matches how the account was
|
||||
provisioned.
|
||||
2. **The per-source size cap.** `limits[3]` (e.g. `500000`) is the per-source size limit; it matches
|
||||
the Enterprise "500,000 words per source" figure. `notebook_limit` = `limits[1]`, `source_limit`
|
||||
= `limits[2]`, `tier` = `limits[4]`. See [rpc-reference.md](rpc-reference.md).
|
||||
|
||||
---
|
||||
|
||||
## Consumer — notebooklm.google.com
|
||||
|
||||
Source: [support.google.com/notebooklm/answer/16213268](https://support.google.com/notebooklm/answer/16213268)
|
||||
· captured **2026-07-09**. Five tiers since May 2026 (Standard, "in Plus" = Google AI Plus, "in Pro"
|
||||
= Google AI Pro, "in Ultra 20 TB", "in Ultra 30 TB"). *"NotebookLM in Plus" ≠ the retired 2024–25
|
||||
"NotebookLM Plus" product.*
|
||||
|
||||
| Feature | Standard (`1`) | Plus (`4`) | Pro (`2`) | Ultra 20TB (`3`) | Ultra 30TB (`6`) |
|
||||
|---|---|---|---|---|---|
|
||||
| Notebooks (per user) | 100 | 200 | 500 | 500 | 500 |
|
||||
| Sources (per notebook) | 50 | 100 | 300 | 500 | 600 |
|
||||
| Chats /day | 50 | 200 | 500 | 2,500 | 5,000 |
|
||||
| Audio Overviews /day | 3 | 6 | 20 | 100 | 200 |
|
||||
| Video Overviews /day | 3 | 6 | 20 | 100 | 200 |
|
||||
| — Cinematic Video /day | — | — | 2 | 10 | 20 |
|
||||
| Reports /day | 10 | 20 | 100 | 500 | 1,000 |
|
||||
| Flashcards /day | 10 | 20 | 100 | 500 | 1,000 |
|
||||
| Quizzes /day | 10 | 20 | 100 | 500 | 1,000 |
|
||||
| Mind Maps /day | 10 | 20 | 100 | 500 | 1,000 |
|
||||
| Deep Research | **10/month** | 3/day | 20/day | 75/day | 200/day |
|
||||
| Data Tables | *Limited* | *More* | *High* | *Higher* | *Highest* |
|
||||
| Infographics | *Limited* | *More* | *High* | *Higher* | *Highest* |
|
||||
| Slide Decks & Revisions | *Limited* | *More* | *High* | *Higher* | *Highest* |
|
||||
| Watermark removal | No | No | No | Yes | Yes (Infographics + Slide Decks) |
|
||||
|
||||
The column headers show the `tier` int this package returns for each plan. *Italic* cells are
|
||||
qualitative labels Google publishes instead of numbers — see
|
||||
[Unquantified features](#unquantified-features).
|
||||
|
||||
## Workspace / EDU
|
||||
|
||||
Source: [support.google.com/notebooklm/answer/16337734](https://support.google.com/notebooklm/answer/16337734)
|
||||
· captured **2026-07-09**. Five access levels mapped to editions: **Standard** (Business Starter,
|
||||
Enterprise Essentials, Frontline, Nonprofits, Education Fundamentals/Standard, …) · **More**
|
||||
(Education Plus, Teaching & Learning add-on) · **Higher** (Business Standard/Plus, Enterprise
|
||||
Standard/Plus, Google AI Pro for Education) · **Expanded** (AI Expanded Access add-on) · **Highest**
|
||||
(AI Ultra Access add-on).
|
||||
|
||||
| Feature | Standard | More | Higher | Expanded | Highest |
|
||||
|---|---|---|---|---|---|
|
||||
| Notebooks (per user) | 100 | 200 | 500 | 500 | 500 |
|
||||
| Sources (per notebook) | 50 | 100 | 300 | **400** | 600 |
|
||||
| Chats /day | 50 | 200 | 500 | **1,000** | 5,000 |
|
||||
| Audio Overviews /day | 3 | 6 | 20 | **40** | 200 |
|
||||
| Video Overviews /day | 3 | 6 | 20 | **40** | 200 |
|
||||
| — Cinematic Video /day | — | — | 2 | **4** | 20 |
|
||||
| Reports / Flashcards / Quizzes / Mind Maps /day (each) | 10 | 20 | 100 | **200** | 1,000 |
|
||||
| Deep Research | **10/month** | 3/day | 20/day | **30/day** | 200/day |
|
||||
| Data Tables / Infographics / Slide Decks | *Limited* | *More* | *Higher* | *Expanded* | *Highest* |
|
||||
| Watermark removal | No | No | No | Yes | Yes (Infographics + Slide Decks) |
|
||||
|
||||
The Workspace ladder deliberately diverges from consumer at the **Expanded** rung (bold): weaker than
|
||||
consumer Ultra-20TB (400 vs 500 sources, 1K vs 2.5K chats, 40 vs 100 audio/video, 200 vs 500 assets,
|
||||
30 vs 75 Deep Research). The `tier=5` ("Expanded") the badge bundle exposes lines up with this level.
|
||||
|
||||
## Enterprise (Gemini Enterprise / Agentspace)
|
||||
|
||||
Source: [docs.cloud.google.com … notebooklm-enterprise/overview](https://docs.cloud.google.com/gemini/enterprise/notebooklm-enterprise/docs/overview)
|
||||
· page stamped **2026-07-07**.
|
||||
|
||||
| Feature | Limit |
|
||||
|---|---|
|
||||
| Notebooks | 500 /user |
|
||||
| Sources | 300 /notebook |
|
||||
| Source size | **500 MB or 500,000 words per source** (only tier with a documented per-source cap) |
|
||||
| Chats | 500 /user/day |
|
||||
| Audio Overviews / Video Overviews / Mind Maps / Reports | 20 /user/day each |
|
||||
| Slide Decks / Infographics | **15 /user/day each** (the only surface publishing numbers for these) |
|
||||
| Flashcards · Quizzes · Deep Research · Data Tables · narrated slideshows | no published quota |
|
||||
|
||||
---
|
||||
|
||||
## Unquantified features
|
||||
|
||||
For **Data Tables, Infographics, and Slide Decks & Revisions**, Google publishes qualitative labels
|
||||
(*Limited / More / High / Higher / Highest*) rather than numbers on consumer and Workspace tiers —
|
||||
and Wayback snapshots (Nov 2025 → Jul 2026) confirm the numbers **never existed**, they were
|
||||
qualitative from the day the rows appeared. Best-known values, with an evidence class per cell:
|
||||
|
||||
| Feature | Tier | Best-known limit | Evidence |
|
||||
|---|---|---|---|
|
||||
| Infographics | Free/Standard | ~3/day | **observed** — community/third-party testing (tenorshare, 2026-05-26); in-app error *"You have reached your daily Infographic limits"*. No official number. |
|
||||
| Infographics | Plus / Pro / Ultra / all Workspace | unknown | qualitative labels only |
|
||||
| Infographics | Enterprise | 15/user/day | **official** (cloud docs) |
|
||||
| Slide Decks & Revisions | consumer / Workspace | unknown | [help page](https://support.google.com/notebooklm/answer/16757456) confirms revision quotas *exist* but publishes no value |
|
||||
| Slide Decks & Revisions | Enterprise | 15/user/day | **official** (cloud docs) |
|
||||
| Data Tables | any tier, incl. Enterprise | unknown | no number in any source; the Enterprise table has no Data Tables row at all |
|
||||
| Narrated slideshows | any tier | — | no separate quota row; subsumed under Slide Decks |
|
||||
|
||||
**Evidence classes:** *official* = published by Google · *observed* = community/third-party report or
|
||||
in-app error text, no official figure · *unknown* = acknowledged-to-exist but unpublished.
|
||||
|
||||
---
|
||||
|
||||
## Reset & counting semantics
|
||||
|
||||
- **Daily quotas reset after 24 hours; monthly quotas after 30 days.** Rolling windows (from first
|
||||
use), not calendar-midnight.
|
||||
- **Notebook** caps are **per-user**; **source** caps are **per-notebook**; all chat/generation
|
||||
quotas are **per-user per-day**. *"Sharing a notebook does not change the source limit for any
|
||||
collaborator."*
|
||||
- **Deep Research is the only monthly quota**, and only at Standard/free (10/month) — a
|
||||
counterintuitive kink (Standard 10/month vs Plus 3/day) Google publishes on both consumer and
|
||||
Workspace pages.
|
||||
- **First-add auto-generated artifacts** (the report/flashcards/infographic/slide-deck/audio/video
|
||||
overview generated when sources are first added) are generated once and **do not count** against
|
||||
limits.
|
||||
|
||||
## Caveats
|
||||
|
||||
- **Dated snapshot.** Consumer tiers were restructured May 2026 (4 tiers → 5, Ultra split into
|
||||
20 TB / 30 TB). Re-check the source links before trusting a value.
|
||||
- **Enforced limits change without doc updates.** On 2025-11-25 NotebookLM temporarily rolled
|
||||
Infographics/Slide Decks back to **0 for Free users** and imposed unspecified extra limits on Pro,
|
||||
due to demand ([@NotebookLM](https://x.com/NotebookLM/status/1993387297081827346)); access was
|
||||
later restored. The published tables did not reflect this.
|
||||
- **No per-source word/size cap is published for consumer/Workspace** — the "500K words/source"
|
||||
figure is documented only for Enterprise. (`limits[3]` ≈ `500000` in the settings vector may be the
|
||||
same cap surfaced per-account; unconfirmed.)
|
||||
- **No sharing/collaborator-count limits** are documented on any tier.
|
||||
|
||||
## Why this lives in docs, not code
|
||||
|
||||
This package ships the **authoritative live signal** as code — `AccountLimits.tier` (the opaque int
|
||||
from the quota block). It deliberately does **not** ship this table as constants, because the data is:
|
||||
|
||||
1. **Heterogeneous** — a mix of official numbers, community-observed values, and
|
||||
acknowledged-but-unknown quotas. `Optional[int]` + provenance could model it, but a typed lookup
|
||||
would imply a false authority the source data doesn't have.
|
||||
2. **Google-owned and volatile** — the tier scheme itself changed in May 2026, and enforced limits
|
||||
have changed without the published tables updating. Baking it into code guarantees stale,
|
||||
authoritative-looking values.
|
||||
3. **Not needed at runtime** — the library never enforces these limits; the server surfaces the
|
||||
`tier` int and the actual `notebook_limit` / `source_limit` the account reports live. Callers that
|
||||
want a human answer look here.
|
||||
|
||||
This mirrors the same call made for the tier→label mapping: ship the int, keep the table in docs.
|
||||
|
||||
## Sources
|
||||
|
||||
- [Consumer limits](https://support.google.com/notebooklm/answer/16213268) (live + 5 Wayback snapshots Nov 2025–Jul 2026)
|
||||
- [Workspace / school limits](https://support.google.com/notebooklm/answer/16337734) (EN + JA)
|
||||
- [Enterprise limits](https://docs.cloud.google.com/gemini/enterprise/notebooklm-enterprise/docs/overview)
|
||||
- [Slide Deck revision help page](https://support.google.com/notebooklm/answer/16757456)
|
||||
- [Create a notebook — usage-limit note](https://support.google.com/notebooklm/answer/16206563)
|
||||
- [@NotebookLM rollback, 2025-11-25](https://x.com/NotebookLM/status/1993387297081827346) · [XDA coverage](https://www.xda-developers.com/google-rolls-back-notebooklms-slide-decks-infographics/)
|
||||
- [Data Tables launch](https://workspaceupdates.googleblog.com/2025/12/transform-sources-structured-data-tables-notebooklm.html) · [blog.google](https://blog.google/technology/google-labs/notebooklm-data-tables/)
|
||||
- [tenorshare infographic-limit report](https://www.tenorshare.ai/ai-tips/notebooklm-infographic-limit.html)
|
||||
|
||||
Research provenance: multi-angle deep-research runs on 2026-07-09 and 2026-07-10 (adversarial
|
||||
verification; Wayback archaeology; EN + JA locales), recorded in
|
||||
[#1825](https://github.com/teng-lin/notebooklm-py/issues/1825).
|
||||
@@ -0,0 +1,439 @@
|
||||
# Tier 12 / Tier 13 Refactor — Historical Record
|
||||
|
||||
> **Status:** Shipped in v0.5.0.
|
||||
> **Current runtime shape:** [`docs/architecture.md`](architecture.md).
|
||||
> **Ratifying decision:** [ADR-0013 — composable session capabilities](adr/0013-composable-session-capabilities.md), which supersedes [ADR-0010 — Session/Kernel split](adr/0010-session-kernel-split.md).
|
||||
> **Last updated:** 2026-06-11
|
||||
|
||||
> **⚠️ Superseded in part by [ADR-0014](adr/0014-feature-local-runtime-adapters.md).**
|
||||
> Later refactors removed the `Session` facade class entirely and then promoted
|
||||
> the flat `_session_*` / `_runtime_*` helpers into the current package homes:
|
||||
> `_runtime/contracts.py`, `_runtime/auth.py`, `_runtime/lifecycle.py`,
|
||||
> `_runtime/config.py`, `_runtime/helpers.py`, `_runtime/transport.py`,
|
||||
> `_runtime/init.py`, and `_middleware/*`. The module names in the migration
|
||||
> tables below are the **v0.5.0 as-shipped** names, not the current tree — see
|
||||
> [`docs/architecture.md`](architecture.md) for the current module map.
|
||||
|
||||
This document is the historical record of the Tier 12 / Tier 13 refactor arc.
|
||||
It exists for two audiences:
|
||||
|
||||
- **Maintainers** who want to understand the design intent — why the
|
||||
feature-facing `Session` boundary was rebuilt around composable
|
||||
capability Protocols rather than a single broad Protocol — and
|
||||
what tradeoffs that locked in.
|
||||
- **First-party callers and test suites** that imported underscore-prefixed
|
||||
(`notebooklm._...`) internals during Tier 11 or earlier, and need
|
||||
to migrate to the new module paths.
|
||||
|
||||
If your code only uses the documented public API
|
||||
([`docs/python-api.md`](python-api.md)), nothing changed for you — the
|
||||
v0.4.1 public surface was a binding constraint on the whole arc.
|
||||
|
||||
## Background
|
||||
|
||||
Tier 12 ("middleware chain") and Tier 13 ("Session/Kernel split")
|
||||
restructured the **private** core layer of `notebooklm-py`:
|
||||
|
||||
- **Tier 12** extracted the authed HTTP path into a composable
|
||||
middleware chain (`AuthedHttpClient` Protocol, `Middleware` Protocol,
|
||||
`RequestContext`). Tracing, metrics, drain bookkeeping, error
|
||||
injection, 429 / 5xx retry, 401 auth-refresh, and the global RPC
|
||||
concurrency semaphore moved into discrete middleware modules and
|
||||
off of the monolithic transport class.
|
||||
- **Tier 13** split the `ClientCore` god-object into `Session` (the
|
||||
orchestrator) and `Kernel` (the pure transport core — owns the
|
||||
`httpx.AsyncClient` and the cookie jar). It also renamed every
|
||||
`_core_*` module out of the legacy `_core` namespace into final
|
||||
homes that later consolidated under `_runtime/`, `_middleware/`, `_chat/`,
|
||||
`_request_types`, `_transport_errors`, `_streaming_post`, `_rpc_executor`,
|
||||
and so on.
|
||||
- The **capability refactor** that followed (ADR-0013) replaced the
|
||||
broad `Session` Protocol that Tier 13 originally shipped with a
|
||||
composable set of narrow capability Protocols
|
||||
(`RpcCaller`, `LoopGuard`, `OperationScopeProvider`,
|
||||
`AsyncWorkRuntime`) plus feature-local runtime Protocols
|
||||
(`ChatRuntime`, `ArtifactsRuntime`, `UploadRuntime`). Feature APIs
|
||||
now depend on the narrowest slice of capability they actually use,
|
||||
not on the orchestrator class.
|
||||
Later ADR-0014/#1327 cleanup tightened this further: the current code
|
||||
passes each concrete collaborator directly, keeps `AuthMetadata` and
|
||||
`OperationScopeProvider` local to their only consumers, and leaves only
|
||||
`Kernel`, `RpcCaller`, and `LoopGuard` in `_runtime/contracts.py`.
|
||||
|
||||
The public `NotebookLMClient` API — `client.notebooks.*`,
|
||||
`client.sources.*`, `client.chat.*`, `client.artifacts.*`, and
|
||||
friends — was preserved across all three movements.
|
||||
|
||||
## What changed by audience
|
||||
|
||||
### Downstream callers using the public API
|
||||
|
||||
Nothing. Every method, property, and attribute reachable through
|
||||
`NotebookLMClient` at v0.4.1 still works with the same signature and
|
||||
return type:
|
||||
|
||||
```python
|
||||
from notebooklm import NotebookLMClient, AuthTokens
|
||||
from notebooklm.rpc import RPCMethod
|
||||
|
||||
async with NotebookLMClient.from_storage() as client:
|
||||
notebooks = await client.notebooks.list()
|
||||
await client.sources.add_url(notebook_id, url)
|
||||
result = await client.chat.ask(notebook_id, question)
|
||||
status = await client.artifacts.generate_audio(notebook_id)
|
||||
```
|
||||
|
||||
One additive surface from this arc was
|
||||
`client.chat.save_answer_as_note(...)`, which is now the canonical
|
||||
citation-rich saved-from-chat workflow. The transitional
|
||||
`client.notes.create_from_chat(...)` forwarder was removed in v0.7.0.
|
||||
|
||||
See [`docs/stability.md`](stability.md) for the public stability
|
||||
contract.
|
||||
|
||||
### First-party callers and test suites
|
||||
|
||||
Underscore-prefixed names are **not** part of the documented public
|
||||
surface and may move again in future tiers. The tables below are
|
||||
provided as a courtesy for first-party callers and test suites that
|
||||
imported them during Tier 11 or earlier.
|
||||
|
||||
**Quick guidance:**
|
||||
|
||||
- The `notebooklm._core` compatibility shim and the
|
||||
`NotebookLMClient._core` attribute alias were both deleted in
|
||||
Phase 4 (#889). Downstream code and tests must import from the
|
||||
current owning modules documented in `docs/architecture.md`.
|
||||
- Prefer the public surface — `notebooklm.NotebookLMClient`,
|
||||
`notebooklm.AuthTokens`, `notebooklm.rpc.RPCMethod`, and the
|
||||
types / exceptions re-exported from the top-level package.
|
||||
- Feature APIs (`NotebooksAPI`, `SourcesAPI`, `ArtifactsAPI`,
|
||||
`ChatAPI`, `ResearchAPI`, `NotesAPI`, `MindMapsAPI`, `SharingAPI`,
|
||||
`SettingsAPI`, `LabelsAPI`)
|
||||
depend on the **narrow capability Protocols** in
|
||||
`notebooklm._runtime.contracts` (`Kernel`, `RpcCaller`,
|
||||
`LoopGuard`) or on single-consumer Protocols defined in their owning
|
||||
modules — not on the deleted concrete `Session` class and not on a
|
||||
broad `Session` Protocol.
|
||||
|
||||
#### Renamed modules
|
||||
|
||||
The Tier 12 `_core_*` module paths no longer resolve.
|
||||
`import notebooklm._core_auth` (and every other row below) now
|
||||
raises `ImportError`. Update your imports to the new path on the
|
||||
right:
|
||||
|
||||
| Tier 12 path | Tier 13 path |
|
||||
|---|---|
|
||||
| `notebooklm._core_auth` | `notebooklm._runtime.auth` |
|
||||
| `notebooklm._core_cache` | `notebooklm._conversation_cache` |
|
||||
| `notebooklm._core_constants` | `notebooklm._runtime.config` |
|
||||
| `notebooklm._core_cookie_persistence` | `notebooklm._cookie_persistence` |
|
||||
| `notebooklm._core_drain` | `notebooklm._transport_drain` |
|
||||
| `notebooklm._core_error_injection` | `notebooklm._error_injection` |
|
||||
| `notebooklm._core_helpers` | `notebooklm._runtime.helpers` |
|
||||
| `notebooklm._core_lifecycle` | `notebooklm._runtime.lifecycle` |
|
||||
| `notebooklm._core_metrics` | `notebooklm._client_metrics` |
|
||||
| `notebooklm._core_polling` | `notebooklm._polling_registry` |
|
||||
| `notebooklm._core_reqid` | `notebooklm._reqid_counter` |
|
||||
| `notebooklm._core_rpc` | `notebooklm._rpc_executor` |
|
||||
| `notebooklm._core_transport` | `notebooklm._request_types` / `notebooklm._transport_errors` / `notebooklm._streaming_post` |
|
||||
|
||||
`notebooklm._core` itself has been deleted.
|
||||
|
||||
#### Moved and renamed symbols
|
||||
|
||||
The Session/Kernel split introduced new home modules for the
|
||||
orchestrator and its transport collaborator. Existing helper names did
|
||||
not change, only their home module did.
|
||||
|
||||
| Tier 12 symbol | Tier 13 home | Notes |
|
||||
|---|---|---|
|
||||
| `notebooklm._core.ClientCore` (class) | Direct collaborators + `notebooklm._runtime.contracts` | `ClientCore` was retired. Feature APIs accept direct collaborators or the narrow shared Protocols in `notebooklm._runtime.contracts` (`Kernel`, `RpcCaller`, `LoopGuard`); the broad `Session` Protocol and `_session_contracts` module were retired — see ADR-0013. |
|
||||
| `notebooklm._core.MAX_RETRY_AFTER_SECONDS` | `notebooklm._transport_errors.MAX_RETRY_AFTER_SECONDS` | No longer re-exported via `_session` or `_core`. |
|
||||
| `notebooklm._core.DEFAULT_*` (timeouts, concurrency knobs) | `notebooklm._runtime.config.DEFAULT_*` | |
|
||||
| `notebooklm._core.AUTH_ERROR_PATTERNS`, `notebooklm._core.is_auth_error` | `notebooklm._runtime.helpers` | |
|
||||
| `notebooklm._core.ERROR_INJECT_ENV_VAR` | `notebooklm._error_injection.ERROR_INJECT_ENV_VAR` | |
|
||||
| `notebooklm._core._SyntheticErrorTransport` (class) | _Removed_ | Synthetic-error substitution moved into `notebooklm._middleware.error_injection.ErrorInjectionMiddleware`. The env-var resolver (`_get_error_injection_mode`) and startup guard (`_refuse_synthetic_error_outside_test_context`) survive in `notebooklm._error_injection`. |
|
||||
| `notebooklm._core.AuthRefreshCoordinator` | `notebooklm._runtime.auth.AuthRefreshCoordinator` | Class unchanged; only the home module moved. |
|
||||
| `notebooklm._core.TransportDrainTracker` | `notebooklm._transport_drain.TransportDrainTracker` | Same. |
|
||||
| `notebooklm._core.ClientMetrics` | `notebooklm._client_metrics.ClientMetrics` | Same. |
|
||||
| `notebooklm._core.ReqidCounter` | `notebooklm._reqid_counter.ReqidCounter` | Same. |
|
||||
| `notebooklm._core.CookiePersistence` | `notebooklm._cookie_persistence.CookiePersistence` | Same. |
|
||||
| `notebooklm._core.ClientLifecycle` | `notebooklm._runtime.lifecycle.ClientLifecycle` | Same. |
|
||||
| `notebooklm._core.RpcExecutor` | `notebooklm._rpc_executor.RpcExecutor` | Same. |
|
||||
| `notebooklm._core` authed transport helpers | `notebooklm._request_types` + `notebooklm._transport_errors` + `notebooklm._streaming_post` | The interim authed-transport Adapter and catch-all helper module were retired; request types, transport errors, and streaming POST behavior now have separate owning modules. |
|
||||
|
||||
#### New modules introduced by Tier 12 / 13
|
||||
|
||||
These modules did not exist before Tier 12 began:
|
||||
|
||||
| Module | Purpose |
|
||||
|---|---|
|
||||
| `notebooklm._session_contracts` | Historical v0.5.0 home for `AuthMetadata`, `Kernel`, and the shared capability Protocols (`RpcCaller`, `LoopGuard`, `OperationScopeProvider`, `AsyncWorkRuntime`) added in the capability refactor (ADR-0013). Current home: `notebooklm._runtime.contracts`, exporting only `Kernel`, `RpcCaller`, and `LoopGuard`; `AuthMetadata` is local to `_source/upload.py`, `OperationScopeProvider` is local to `_artifact/polling.py`, and `AsyncWorkRuntime` was deleted. |
|
||||
| `notebooklm._kernel` | Concrete `Kernel` transport core (owns the `httpx.AsyncClient`, exposes `post` / `cookies` / `aclose`). Located at root (`src/notebooklm/_kernel.py`), not nested. |
|
||||
| `notebooklm._middleware` | Middleware chain primitives (`Middleware` Protocol, `NextCall` callable type, `RpcRequest` / `RpcResponse` envelope dataclasses, `build_chain` composer). |
|
||||
| `notebooklm._middleware_tracing` | Tier 12 PR 12.3 — request tracing middleware. |
|
||||
| `notebooklm._middleware_metrics` | Tier 12 PR 12.4 — metrics collection middleware. |
|
||||
| `notebooklm._middleware_drain` | Tier 12 PR 12.5 — drain bookkeeping middleware. |
|
||||
| `notebooklm._middleware.error_injection` | Test-only error-injection middleware. |
|
||||
| `notebooklm._middleware_retry` | Tier 12 PR 12.7 — 429 / 5xx retry middleware. |
|
||||
| `notebooklm._middleware_auth_refresh` | Tier 12 PR 12.8 — auth-refresh-on-401 middleware. |
|
||||
| `notebooklm._middleware_semaphore` | Tier 12 PR 12.9 — global RPC concurrency cap. |
|
||||
| `notebooklm._chat.transport` | Chat-domain consumer-side error mapping over the shared authed POST pipeline. Replaces the chat-side wrapper that previously lived on `_core.rpc_call`. |
|
||||
| `notebooklm._transport_errors` | Terminal `Kernel.post` error mapping into transport exceptions consumed by retry/auth middleware. |
|
||||
| `notebooklm._request_types` | Shared dataclasses + type aliases for authed-POST request construction: `AuthSnapshot`, `BuildRequest`, `PostBody`, and `BuildRequestResult`. |
|
||||
| `notebooklm._streaming_post` | Size-capped streaming POST helper used by `Kernel.post`. |
|
||||
|
||||
#### Deleted symbols and changed defaults
|
||||
|
||||
| Symbol or default | Replacement / new behavior |
|
||||
|---|---|
|
||||
| `notebooklm._core._SyntheticErrorTransport` (deleted) | `notebooklm._middleware.error_injection.ErrorInjectionMiddleware` (chain-resident; mode is still resolved from `NOTEBOOKLM_VCR_RECORD_ERRORS` via `_error_injection._get_error_injection_mode`). |
|
||||
| Strict-decode soft mode | Strict decoding is now the only mode. The legacy lenient opt-out was removed in v0.7.0; setting old lenient-decode environment toggles no longer restores legacy behavior. See [ADR-0011](adr/0011-schema-validation-policy.md). |
|
||||
|
||||
## Design intent
|
||||
|
||||
The capability refactor (ADR-0013) replaced the broad
|
||||
`_session_contracts.Session` Protocol that Tier 13 originally shipped.
|
||||
That Protocol had become a capability bag:
|
||||
|
||||
```python
|
||||
class Session(Protocol):
|
||||
auth: AuthMetadata
|
||||
kernel: Kernel
|
||||
async def rpc_call(...) -> Any: ...
|
||||
async def transport_post(...) -> httpx.Response: ...
|
||||
async def next_reqid(...) -> int: ...
|
||||
def assert_bound_loop(self) -> None: ...
|
||||
def operation_scope(self, label: str) -> AbstractAsyncContextManager[None]: ...
|
||||
def register_drain_hook(self, name, hook) -> None: ...
|
||||
```
|
||||
|
||||
That shape violated the intent of
|
||||
[ADR-0010](adr/0010-session-kernel-split.md): feature APIs should not
|
||||
depend on concrete `Session` internals. It also made dependencies
|
||||
hard to read — most features only needed logical RPC calls, while
|
||||
chat, uploads, and artifact polling needed narrower specialized
|
||||
runtime slices.
|
||||
|
||||
### Design rules
|
||||
|
||||
The capability model is built on six rules:
|
||||
|
||||
1. Promote a capability to `_runtime/contracts.py` only when it is
|
||||
shared by more than one feature or service, except for `Kernel`,
|
||||
which is the typed transport surface of the concrete client-owned
|
||||
kernel.
|
||||
2. Keep single-feature runtime needs local to the owning feature
|
||||
module.
|
||||
3. Feature-facing Protocols must not advertise unrelated capabilities
|
||||
from a broad host object.
|
||||
4. Prefer feature-owned collaborators over widening shared session
|
||||
contracts.
|
||||
5. Remove old `core` vocabulary from touched feature APIs.
|
||||
6. Do not use mixins for dependency expression. Use Protocols for
|
||||
required capabilities and collaborators / services for extracted
|
||||
behavior.
|
||||
|
||||
### Shared capability Protocols
|
||||
|
||||
`src/notebooklm/_session_contracts.py` originally ended up containing
|
||||
only shared capability Protocols:
|
||||
|
||||
```python
|
||||
class RpcCaller(Protocol):
|
||||
async def rpc_call(
|
||||
self,
|
||||
method: RPCMethod,
|
||||
params: list[Any],
|
||||
source_path: str = "/",
|
||||
allow_null: bool = False,
|
||||
_is_retry: bool = False,
|
||||
*,
|
||||
disable_internal_retries: bool = False,
|
||||
operation_variant: str | None = None,
|
||||
) -> Any: ...
|
||||
|
||||
|
||||
class LoopGuard(Protocol):
|
||||
def assert_bound_loop(self) -> None: ...
|
||||
|
||||
|
||||
class OperationScopeProvider(Protocol):
|
||||
def operation_scope(self, label: str) -> AbstractAsyncContextManager[None]: ...
|
||||
|
||||
|
||||
class AsyncWorkRuntime(LoopGuard, OperationScopeProvider, Protocol):
|
||||
"""Runtime support for feature-owned async work."""
|
||||
```
|
||||
|
||||
Current code has tightened that further: `_runtime/contracts.py`
|
||||
exports `Kernel`, `RpcCaller`, and `LoopGuard`; `OperationScopeProvider`
|
||||
is local to `_artifact/polling.py`; `AuthMetadata` is local to
|
||||
`_source/upload.py`; `AsyncWorkRuntime` was removed.
|
||||
|
||||
The following were **not** globally promoted:
|
||||
|
||||
- `auth`, `kernel`, `transport_post(...)`, `next_reqid(...)` —
|
||||
needed only by uploads and chat respectively. They live on the
|
||||
concrete `Session` and on feature-local runtimes that consume
|
||||
them, not on a shared Protocol.
|
||||
- `register_drain_hook(...)` — kept on the concrete drain collaborator
|
||||
passed into `ArtifactsAPI`, since artifact polling is the only behavior
|
||||
that registers close-time feature cleanup.
|
||||
|
||||
### Feature-local runtimes
|
||||
|
||||
For part of the migration, features that needed a specialised slice of
|
||||
runtime capability declared that slice as a Protocol in the feature's
|
||||
own module:
|
||||
|
||||
- `ChatRuntime` in `_chat.py` — `RpcCaller + LoopGuard +
|
||||
transport_post + next_reqid`. `transport_post` is consumed via
|
||||
`chat_aware_authed_post(...)` in `_chat/transport.py`.
|
||||
- `ArtifactsRuntime` in `_artifacts.py` — `RpcCaller +
|
||||
AsyncWorkRuntime + DrainHookRegistration`. Artifact polling owns
|
||||
the only close-time cleanup hook in the codebase today.
|
||||
- `UploadRuntime` in `_source_upload.py` — `RpcCaller +
|
||||
OperationScopeProvider + LoopGuard`. The upload pipeline also receives
|
||||
`kernel` and `auth` as constructor args, since uploads need
|
||||
upload-specific auth routing and live cookies — but those wires
|
||||
are explicit, not implicit through a god-object.
|
||||
|
||||
If a future feature ever needs the same local slice that another
|
||||
feature has carved out, the rule is: promote the Protocol to
|
||||
`_runtime/contracts.py` only when there is a real second consumer,
|
||||
not on speculation.
|
||||
|
||||
Those composite runtime Protocols were later removed. Current
|
||||
constructors take the direct collaborators they need by keyword:
|
||||
`ArtifactsAPI(rpc=..., drain=..., lifecycle=...)`,
|
||||
`ChatAPI(rpc=..., transport=..., reqid=..., loop_guard=...)`, and
|
||||
`SourceUploadPipeline(rpc=..., drain=..., lifecycle=..., kernel=...,
|
||||
auth=...)`.
|
||||
|
||||
### Note / mind-map service split
|
||||
|
||||
The capability refactor also separated note-row primitives from
|
||||
mind-map product behavior. Before the split, `_mind_map.py` mixed
|
||||
three responsibilities:
|
||||
|
||||
1. Backend row classification (note vs. mind-map vs. saved-chat).
|
||||
2. Mind-map filtering and content extraction.
|
||||
3. Saved-from-chat note encoding (chat citations + rich source
|
||||
passage anchors).
|
||||
|
||||
These were factored apart:
|
||||
|
||||
- **`_note_service.py`** owns the generic note row primitives:
|
||||
`fetch_note_rows`, `classify_row` (over a private `NoteRowKind`
|
||||
enum), `extract_content`, `create_note`, `update_note`,
|
||||
`delete_note`. Mind maps are stored in the same note-row
|
||||
collection that NotebookLM returns from `GET_NOTES_AND_MIND_MAPS`,
|
||||
so note-row classification is the right place to recognise them.
|
||||
- **`_mind_map.py`** kept the mind-map boundary as
|
||||
`NoteBackedMindMapService` — `list_mind_maps`,
|
||||
`extract_content`, `delete_mind_map` — delegating to
|
||||
`NoteService` for the underlying row operations.
|
||||
`NotesAPI.list_mind_maps(...)` and `NotesAPI.delete_mind_map(...)`
|
||||
forward through this service, preserving their public signatures.
|
||||
- **Saved-from-chat note encoding** moved out of `_mind_map.py` and
|
||||
into `_chat/notes.py`, where `ChatAPI.save_answer_as_note(...)`
|
||||
owns the workflow. `NotesAPI.create_from_chat(...)` was a deprecated
|
||||
forwarder during the migration window and was removed in v0.7.0.
|
||||
|
||||
`NoteRowKind` stays private — it is an internal classification of
|
||||
rows returned by the undocumented `GET_NOTES_AND_MIND_MAPS` RPC, not
|
||||
a public API type.
|
||||
|
||||
### Constructor and naming rules
|
||||
|
||||
Feature APIs adopted consistent dependency-naming conventions:
|
||||
|
||||
- Pure-RPC features store `self._rpc`. Constructor parameter is
|
||||
named `rpc: RpcCaller`.
|
||||
- Features with multiple runtime needs take direct keyword-only
|
||||
collaborators and store only the fields they use.
|
||||
- The old `self._core` vocabulary was removed from every touched
|
||||
feature module.
|
||||
- Extra collaborators are keyword-only:
|
||||
|
||||
```python
|
||||
SourcesAPI(rpc, *, uploader=source_uploader)
|
||||
NotebooksAPI(rpc, *, sources_api=sources)
|
||||
ChatAPI(rpc=rpc, transport=transport, reqid=reqid, loop_guard=lifecycle,
|
||||
notebooks=notebooks)
|
||||
ArtifactsAPI(rpc=rpc, drain=drain, lifecycle=lifecycle,
|
||||
notebooks=notebooks, mind_maps=mind_maps,
|
||||
note_service=note_service)
|
||||
NotesAPI(*, notes=note_service, mind_maps=mind_maps)
|
||||
MindMapsAPI(rpc=rpc, mind_maps=mind_maps, artifacts=artifacts,
|
||||
notebooks=notebooks)
|
||||
LabelsAPI(rpc, list_sources=sources.list)
|
||||
```
|
||||
|
||||
Compatibility aliases and fallback constructors that read missing
|
||||
collaborators off a broad session were removed — every feature now
|
||||
requires its dependencies explicitly. `_client_assembly.py::_assemble_client`
|
||||
is the single wiring root that hands them out.
|
||||
|
||||
## How it landed
|
||||
|
||||
The refactor arc shipped in four phases against the
|
||||
`refactor-completion-plan` series (PRs in the #84x–#92x range).
|
||||
Detailed phase notes live in the merged PR descriptions; the
|
||||
high-level shape was:
|
||||
|
||||
- **Phase 1 — additive contracts.** New capability Protocols
|
||||
(`RpcCaller`, `LoopGuard`, `OperationScopeProvider`,
|
||||
`AsyncWorkRuntime`) were added to `_session_contracts.py`
|
||||
alongside the existing broad `Session` Protocol. Nothing was
|
||||
removed. Build stayed green.
|
||||
- **Phase 2 — pure-RPC feature retyping.** `NotebooksAPI`,
|
||||
`ResearchAPI`, `SettingsAPI`, `SharingAPI` were retyped to
|
||||
`RpcCaller`, with `self._core` renamed to `self._rpc`. The
|
||||
`FakeSession` test fixture shrunk in the same commits to match
|
||||
the narrower contract.
|
||||
- **Phase 3 — feature-local runtimes.** `ChatRuntime`,
|
||||
`ArtifactsRuntime`, and `UploadRuntime` were introduced in their
|
||||
owning modules. `ChatAPI`, `ArtifactsAPI`, `SourcesAPI`, and
|
||||
`SourceUploadPipeline` migrated to the new local Protocols.
|
||||
Compatibility fallbacks (the `drain_hooks=None -> session`
|
||||
fallback in `ArtifactsAPI`, the `session.kernel` / `session.auth`
|
||||
/ `record_upload_queue_wait` fallbacks in `SourcesAPI`, the
|
||||
`core=` alias on `ChatAPI`) were removed in the same commits
|
||||
that updated their direct-construction test sites.
|
||||
- **Phase 4 — note / mind-map split + capability cleanup.**
|
||||
`NoteService` and `NoteBackedMindMapService` were introduced,
|
||||
artifact generation and download paths were rewired through them,
|
||||
`ChatAPI.save_answer_as_note(...)` was added, and
|
||||
`NotesAPI.create_from_chat(...)` was converted to a deprecated
|
||||
forwarder (later removed in v0.7.0). The module-level `_mind_map`
|
||||
wrappers were removed.
|
||||
The broad `Session` Protocol was deleted from
|
||||
`_session_contracts.py`, along with the broad `FakeSession`
|
||||
defaults shape and the broad-Protocol test pin. The `_core.py`
|
||||
compatibility shim and the `NotebookLMClient._core` attribute
|
||||
alias were both removed in this phase (#889).
|
||||
|
||||
Implementation tactics — line-number-specific constructor changes,
|
||||
the ordering rules for compatibility-fallback removals, the
|
||||
`_source_upload.py` local `RpcCaller` → `RpcCallback` rename to
|
||||
avoid name collision with the new shared Protocol — are recorded in
|
||||
the merged PR descriptions, not in this document. The post-refactor
|
||||
runtime shape is canonicalized in [`docs/architecture.md`](architecture.md).
|
||||
|
||||
## See also
|
||||
|
||||
- [`docs/architecture.md`](architecture.md) — current runtime shape,
|
||||
capability-protocol model, and dispatch path.
|
||||
- [ADR-0010 — Session / Kernel split](adr/0010-session-kernel-split.md) —
|
||||
the design driver for Tier 13. **Superseded by ADR-0013.**
|
||||
- [ADR-0011 — Schema validation policy](adr/0011-schema-validation-policy.md) —
|
||||
the strict-decode default flip.
|
||||
- [ADR-0012 — Implementation-surface convention](adr/0012-implementation-surface-convention.md).
|
||||
- [ADR-0013 — Composable session capabilities](adr/0013-composable-session-capabilities.md) —
|
||||
the ratifying decision for the capability model described here.
|
||||
- [`docs/stability.md`](stability.md) — public stability contract.
|
||||
- [`docs/python-api.md`](python-api.md) — canonical public API reference.
|
||||
@@ -0,0 +1,572 @@
|
||||
# Release Checklist
|
||||
|
||||
**Status:** Active
|
||||
**Last Updated:** 2026-05-23
|
||||
|
||||
Checklist for releasing a new version of `notebooklm-py`.
|
||||
|
||||
> **For Claude Code:** Follow this checklist step by step. **NO STEPS ARE OPTIONAL.** "Quick release" means efficient execution, NOT skipping steps.
|
||||
>
|
||||
> **Critical rules:**
|
||||
> 1. **Always use a worktree** - never work directly on main for releases
|
||||
> 2. **Use PRs, not direct pushes** - all release changes go through a PR
|
||||
> 3. **Explicit confirmation required** for: creating PR, publishing to TestPyPI, creating tags, pushing tags
|
||||
> 4. **"ok" is not confirmation** - restate what you're about to do and wait for explicit "yes"
|
||||
> 5. **TestPyPI is mandatory** - it catches packaging issues that tests cannot detect
|
||||
> 6. **Public API compatibility audit is mandatory** - do not publish while
|
||||
> `scripts/audit_public_api_compat.py` reports unapproved breaks
|
||||
|
||||
---
|
||||
|
||||
## Pre-Flight Summary
|
||||
|
||||
Before starting, present this summary to the user:
|
||||
|
||||
```
|
||||
Release Plan for vX.Y.Z:
|
||||
1. Create release worktree (`release/vX.Y.Z` branch)
|
||||
2. Update pyproject.toml and CHANGELOG.md
|
||||
3. Run public API compatibility audit
|
||||
4. Run pre-commit checks (ruff, mypy, pytest)
|
||||
5. Commit changes
|
||||
6. ⏸️ CONFIRM: Create PR to main?
|
||||
7. Wait for CI to pass on PR
|
||||
8. Run E2E and RPC health checks on release branch
|
||||
9. ⏸️ CONFIRM: Publish to TestPyPI?
|
||||
10. Verify TestPyPI package
|
||||
11. Merge PR to main
|
||||
12. ⏸️ CONFIRM: Create and push tag vX.Y.Z?
|
||||
13. Wait for PyPI publish
|
||||
14. Create GitHub release (add `--prerelease` for a pre-release — see [Pre-releases](#pre-releases-alpha--beta--rc))
|
||||
15. Clean up worktree
|
||||
|
||||
Proceed with release preparation?
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Setup
|
||||
|
||||
### Create Release Worktree
|
||||
|
||||
- [ ] Create a dedicated worktree for the release:
|
||||
```bash
|
||||
git worktree add .worktrees/vX.Y.Z -b release/vX.Y.Z main
|
||||
cd .worktrees/vX.Y.Z
|
||||
```
|
||||
- [ ] Set up the development environment:
|
||||
```bash
|
||||
# Canonical contributor install; add --extra mcp/--extra server when
|
||||
# validating those adapters locally. `[all]` also includes mcp+server
|
||||
# and deliberately excludes cookies (Python 3.13+ rookiepy issue).
|
||||
uv sync --frozen --extra browser --extra dev --extra markdown
|
||||
uv run playwright install chromium
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pre-Release
|
||||
|
||||
### Documentation
|
||||
|
||||
- [ ] Verify README.md reflects current features
|
||||
- [ ] Check CLI reference matches `notebooklm --help` output
|
||||
- [ ] Verify Python API docs match public exports in `__init__.py`
|
||||
- [ ] Update `Last Updated` dates in modified docs
|
||||
- [ ] Verify example scripts have valid syntax:
|
||||
```bash
|
||||
uv run python -m py_compile examples/*.py
|
||||
```
|
||||
|
||||
**Related docs to check/update if relevant:**
|
||||
|
||||
| Doc | Update when... |
|
||||
|-----|----------------|
|
||||
| [README.md](../README.md) | New features, changed capabilities, Beyond the Web UI section |
|
||||
| [SKILL.md](../SKILL.md) | New CLI commands, changed flags, new workflows |
|
||||
| [cli-reference.md](cli-reference.md) | Any CLI changes |
|
||||
| [python-api.md](python-api.md) | New/changed Python API |
|
||||
| [troubleshooting.md](troubleshooting.md) | New known issues, fixed issues to remove |
|
||||
| [development.md](development.md) | Architecture changes, new test patterns |
|
||||
| [configuration.md](configuration.md) | New env vars, config options |
|
||||
| [stability.md](stability.md) | Public API changes, deprecations |
|
||||
|
||||
### Version Bump
|
||||
|
||||
- [ ] Determine version bump type using this decision tree:
|
||||
|
||||
```
|
||||
Did you add new items to `__all__` in `__init__.py`?
|
||||
├── YES → MINOR (new public API)
|
||||
└── NO → PATCH (fixes, logging, UX, internal improvements)
|
||||
|
||||
When in doubt, it's PATCH.
|
||||
```
|
||||
|
||||
For a **pre-release**, target the final version plus a pre-release serial
|
||||
(e.g. `0.8.0a1`) — see [Pre-releases](#pre-releases-alpha--beta--rc).
|
||||
|
||||
See [Version Numbering](#version-numbering) for full details.
|
||||
|
||||
- [ ] Update version in `pyproject.toml`:
|
||||
```toml
|
||||
version = "X.Y.Z"
|
||||
```
|
||||
|
||||
- [ ] Update the matching version in `desktop-extension/manifest.json` (`"version": "X.Y.Z"`).
|
||||
It must equal `pyproject.toml` — `tests/unit/test_mcp_desktop_extension.py` enforces this,
|
||||
and `publish-mcpb.yml` aborts the release-asset build on a mismatch.
|
||||
|
||||
### Public API Compatibility Gate
|
||||
|
||||
- [ ] Run the cross-release public API audit before committing release changes:
|
||||
```bash
|
||||
uv run python scripts/audit_public_api_compat.py
|
||||
```
|
||||
- [ ] Read every reported break. The audit compares this checkout with the
|
||||
latest reachable **stable** release tag (pre-releases are skipped; override
|
||||
with `--baseline-ref vX.Y.Z` only when auditing against a specific previous
|
||||
release).
|
||||
- [ ] If the audit reports an unapproved break, prefer a compatibility shim or
|
||||
restored alias. Do **not** proceed to packaging while unapproved breaks remain.
|
||||
- [ ] If a break is intentional and allowed by the stability policy, update all
|
||||
of these in the same PR:
|
||||
- `docs/stability.md` and `docs/deprecations.md` when the change is a
|
||||
deprecation removal or newly deprecated surface
|
||||
- `CHANGELOG.md` with the migration path
|
||||
- `scripts/api-compat-allowlist.json` with the exact `code`, `object`, and a
|
||||
reviewer-readable reason; use its `extra_public_names` section only for
|
||||
documented names that are intentionally public but not listed in
|
||||
`__all__`
|
||||
- [ ] Re-run the audit. The acceptable release state is either no breaks or
|
||||
only reviewed allowlisted breaks printed by the script.
|
||||
- [ ] If CLI commands, flags, arguments, help text, or env-var bindings changed,
|
||||
also run the CLI contract baseline:
|
||||
```bash
|
||||
uv run pytest tests/unit/cli/test_cli_contract.py
|
||||
```
|
||||
|
||||
The allowlist is not a bypass for accidental breakage. It is a paper trail for
|
||||
intentional removals already permitted by `docs/stability.md`, such as a
|
||||
completed deprecation cycle. Public enum members exposed through documented
|
||||
imports, including `notebooklm.rpc.RPCMethod`, count as API surface. So do
|
||||
documented client namespace methods under `NotebookLMClient.notebooks`,
|
||||
`sources`, `artifacts`, `chat`, `research`, `notes`, `settings`, and
|
||||
`sharing`. Function signatures include positional/keyword compatibility and
|
||||
default values; changing a default is a public behavior change.
|
||||
|
||||
The allowlist is **release-scoped**: each entry records a break pending the
|
||||
*next* stable release, not a permanent exemption. A pre-release tag does not
|
||||
advance the baseline (see the Pre-releases section). Once a stable `vX.Y.Z`
|
||||
ships, its entries are in the baseline and must be pruned — see
|
||||
[Prune the API-Compat Allowlist](#prune-the-api-compat-allowlist). The Code
|
||||
Quality job runs the audit with `--check-stale`, so a stale entry (one matching
|
||||
no break against the baseline) is a CI failure, not silent cruft.
|
||||
|
||||
### Changelog
|
||||
|
||||
- [ ] Get commits since last release:
|
||||
```bash
|
||||
git log $(git describe --tags --abbrev=0)..HEAD --oneline
|
||||
```
|
||||
During a **pre-release cycle**, base the range on the last *stable* tag so the
|
||||
aggregate changelog captures the whole cycle (a plain `git describe` would
|
||||
start at the alpha):
|
||||
```bash
|
||||
git log $(git describe --tags --abbrev=0 --match 'v[0-9]*.[0-9]*.[0-9]*' \
|
||||
--exclude '*a[0-9]*' --exclude '*b[0-9]*' --exclude '*rc[0-9]*')..HEAD --oneline
|
||||
```
|
||||
- [ ] Generate changelog entries in Keep a Changelog format:
|
||||
- **Added** - New features
|
||||
- **Fixed** - Bug fixes
|
||||
- **Changed** - Changes in existing functionality
|
||||
- **Deprecated** - Soon-to-be removed features
|
||||
- **Removed** - Removed features
|
||||
- **Security** - Security fixes
|
||||
- [ ] Add entries under `## [Unreleased]` in `CHANGELOG.md`
|
||||
- [ ] Move `[Unreleased]` content to new version section:
|
||||
```markdown
|
||||
## [Unreleased]
|
||||
|
||||
## [X.Y.Z] - YYYY-MM-DD
|
||||
```
|
||||
- [ ] Update comparison links at bottom of `CHANGELOG.md`:
|
||||
```markdown
|
||||
[Unreleased]: https://github.com/teng-lin/notebooklm-py/compare/vX.Y.Z...HEAD
|
||||
[X.Y.Z]: https://github.com/teng-lin/notebooklm-py/compare/vPREV...vX.Y.Z
|
||||
```
|
||||
|
||||
### Pre-Commit Checks
|
||||
|
||||
- [ ] Run all checks before committing:
|
||||
```bash
|
||||
uv run pre-commit run --all-files && uv run mypy src/notebooklm --ignore-missing-imports && uv run pytest
|
||||
```
|
||||
- [ ] Ensure CI runs the same lint gate (`pre-commit run --all-files`) as local release prep
|
||||
- [ ] Run documentation drift checks (mirror the CI gates in `.github/workflows/test.yml`):
|
||||
```bash
|
||||
uv run python scripts/check_ci_install_parity.py
|
||||
uv run python scripts/check_claude_md_freshness.py
|
||||
uv run python scripts/check_docs_module_refs.py
|
||||
# second run confirms release edits did not introduce new API drift
|
||||
uv run python scripts/audit_public_api_compat.py --check-stale
|
||||
```
|
||||
- [ ] Fix any issues before proceeding
|
||||
|
||||
### Commit
|
||||
|
||||
- [ ] Verify changes:
|
||||
```bash
|
||||
git diff
|
||||
```
|
||||
- [ ] Commit (stage `uv.lock` too — the version bump changes its workspace-package
|
||||
entry, and for a pre-release the `uv sync` re-lock in the Pre-releases section
|
||||
requires it; staging it is a no-op on releases where it did not change):
|
||||
```bash
|
||||
git add pyproject.toml desktop-extension/manifest.json CHANGELOG.md uv.lock docs/
|
||||
git commit -m "chore: release vX.Y.Z"
|
||||
```
|
||||
- [ ] Show commit to user:
|
||||
```bash
|
||||
git show --stat
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI Verification
|
||||
|
||||
### Create Pull Request
|
||||
|
||||
- [ ] **⏸️ CONFIRM:** Ask user "Ready to create PR for release vX.Y.Z?"
|
||||
- [ ] Push branch and create PR:
|
||||
```bash
|
||||
git push -u origin release/vX.Y.Z
|
||||
gh pr create --title "chore: release vX.Y.Z" --body "Release vX.Y.Z
|
||||
|
||||
See CHANGELOG.md for details."
|
||||
```
|
||||
- [ ] Wait for **test.yml** to pass:
|
||||
- Linting and formatting
|
||||
- Type checking
|
||||
- Unit and integration tests (Python 3.10-3.14, all platforms)
|
||||
|
||||
### E2E Tests on Release Branch
|
||||
|
||||
- [ ] Go to **Actions** → **Nightly E2E**
|
||||
- [ ] Click **Run workflow**, set **custom_branch** to `release/vX.Y.Z`
|
||||
- [ ] Wait for E2E tests to pass
|
||||
- [ ] If E2E tests fail:
|
||||
1. Fix issues in the release worktree
|
||||
2. Commit and push
|
||||
3. Re-run E2E tests
|
||||
|
||||
### RPC Health Check on Release Branch
|
||||
|
||||
- [ ] Go to **Actions** → **RPC Health Check**
|
||||
- [ ] Click **Run workflow**, set **custom_branch** to `release/vX.Y.Z`
|
||||
- [ ] Wait for RPC health check to pass
|
||||
- [ ] If RPC health check fails:
|
||||
1. Fix issues in the release worktree
|
||||
2. Commit and push
|
||||
3. Re-run RPC health check
|
||||
|
||||
### MCP connector smoke (manual, per release)
|
||||
|
||||
The nightly E2E run now installs `--extra mcp` and exercises the MCP/CLI layers
|
||||
against the live API (`tests/e2e/test_mcp*.py`, `test_cli_live.py`) — so the
|
||||
on-demand Nightly run above already covers Layer A (tools ⇄ live API) and Layer B
|
||||
(HTTP transport + signed-URL routes, in-process). What it **cannot** cover is
|
||||
claude.ai actually driving the remote connector (OAuth login + the browser
|
||||
upload/download pages). Verify that by hand, once per release (~10 min):
|
||||
|
||||
- [ ] `cd deploy && make dev` (or point at your deployed tunnel) and connect the
|
||||
connector in claude.ai.
|
||||
- [ ] **OAuth login page** renders and the password gate accepts your password.
|
||||
- [ ] List notebooks through the connector.
|
||||
- [ ] Add a URL source.
|
||||
- [ ] Ask a question and get a grounded answer.
|
||||
- [ ] Generate **one** artifact (e.g. a report).
|
||||
- [ ] **Upload** a local file via the signed link (open it in a browser, pick a
|
||||
file, confirm the source lands).
|
||||
- [ ] **Download** the generated artifact via the signed link.
|
||||
|
||||
Bootstrap / sanity helper for the upload+download halves (drives a RUNNING
|
||||
server's file routes, prints PASS/FAIL). Requires the `mcp` extra (e.g.
|
||||
`uv sync --extra mcp`):
|
||||
|
||||
```bash
|
||||
python scripts/mcp_live_smoke.py \
|
||||
--base-url https://your-tunnel.example.com \
|
||||
--bearer "$NOTEBOOKLM_MCP_TOKEN" \
|
||||
--notebook <notebook-id>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Package Verification
|
||||
|
||||
> **⚠️ REQUIRED:** Do NOT skip TestPyPI verification. Always test on TestPyPI before publishing to PyPI. This catches packaging issues that unit tests cannot detect (missing files, broken imports, dependency problems).
|
||||
|
||||
### Publish to TestPyPI
|
||||
|
||||
- [ ] **⏸️ CONFIRM:** Ask user "Ready to publish to TestPyPI?"
|
||||
- [ ] Go to **Actions** → **Publish to TestPyPI**
|
||||
- [ ] Click **Run workflow**, select the **release/vX.Y.Z** branch
|
||||
- [ ] Wait for upload to complete
|
||||
- [ ] Verify package appears: https://test.pypi.org/project/notebooklm-py/
|
||||
|
||||
> **Note:** TestPyPI does not allow re-uploading the same version. If you need to fix issues after publishing, bump the patch version and start over. For a pre-release, bump the **pre-release serial** (`a1 → a2`), not the patch.
|
||||
|
||||
### Verify TestPyPI Package
|
||||
|
||||
- [ ] Go to **Actions** → **Verify Package**
|
||||
- [ ] Click **Run workflow** with **source**: `testpypi`
|
||||
- [ ] Wait for all tests to pass (unit, integration, E2E)
|
||||
- [ ] If verification fails:
|
||||
1. Fix issues in the release worktree
|
||||
2. Bump patch version in `pyproject.toml` (for a pre-release, bump the
|
||||
pre-release serial, e.g. `0.8.0a1 → 0.8.0a2`)
|
||||
3. Update `CHANGELOG.md` with fix
|
||||
4. Commit, push, and re-run **Publish to TestPyPI**
|
||||
|
||||
#### How the verify chain works
|
||||
|
||||
The **Verify Package** workflow (`.github/workflows/verify-package.yml`) exercises a published wheel in two phases so packaging bugs cannot silently fall through to a stale PyPI mirror:
|
||||
|
||||
1. **Dep tree from `uv.lock`.** `uv sync --frozen --extra browser --extra dev --extra markdown --extra mcp --extra server` installs the locked dependency tree for the full non-cookies extra set into `.venv/`: browser automation, developer tooling, Markdown export, MCP, and REST server dependencies. `cookies` stays excluded because of the Python 3.13+ `rookiepy` issue. This produces a deterministic dep tree without any TestPyPI lookups.
|
||||
2. **Wheel from the chosen index, `--no-deps`.** `uv pip install --python .venv/bin/python --no-deps --reinstall --no-cache --only-binary=:all: --index-url <testpypi|pypi> "notebooklm-py==<version>"` swaps the editable install left behind by `uv sync` for the actual published wheel. `--no-deps` is load-bearing: without it the previous `--extra-index-url https://pypi.org/simple/` fallback would mask a broken/missing TestPyPI upload by resolving an older version from PyPI. `--reinstall --no-cache --only-binary=:all:` guarantee we test the freshly-uploaded wheel and never a cached sdist. The explicit `--python .venv/bin/python` is required because `uv sync` does not seed `pip` into the project venv — a bare `source .venv/bin/activate && pip install …` would silently fall back to the runner's system pip and leave the editable install in place.
|
||||
|
||||
The same chain runs for `source: pypi` (post-publish verification) — only the wheel index changes; the locked dep tree is identical.
|
||||
|
||||
The `Publish to PyPI` step in `publish.yml` also opts into **PEP 740 attestations** (`attestations: true`). `pypa/gh-action-pypi-publish` generates an in-toto attestation per uploaded artifact and signs it under Trusted Publishing (OIDC, no API token); PyPI accepts and stores the attestation alongside the wheel, giving downstream consumers cryptographic proof the wheel was built by this GitHub workflow on this tagged commit. The PyPA action enables attestations by default for Trusted Publishing flows, so the explicit `attestations: true` is documentation-as-code rather than a feature flag.
|
||||
|
||||
---
|
||||
|
||||
## Merge to Main
|
||||
|
||||
- [ ] Once TestPyPI verification passes, merge the PR:
|
||||
```bash
|
||||
gh pr merge --squash --delete-branch
|
||||
```
|
||||
- [ ] Pull latest main (in main repo):
|
||||
```bash
|
||||
cd /path/to/notebooklm-py
|
||||
git pull origin main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Release
|
||||
|
||||
### Tag and Publish
|
||||
|
||||
- [ ] **⏸️ CONFIRM:** Ask user "TestPyPI verified. Ready to create tag vX.Y.Z and publish to PyPI? This is irreversible."
|
||||
- [ ] Create tag (on main branch):
|
||||
```bash
|
||||
git tag vX.Y.Z
|
||||
```
|
||||
- [ ] Push tag:
|
||||
```bash
|
||||
git push origin vX.Y.Z
|
||||
```
|
||||
- [ ] Wait for **publish.yml** to complete
|
||||
- [ ] Verify on PyPI: https://pypi.org/project/notebooklm-py/
|
||||
|
||||
### PyPI Verification
|
||||
|
||||
- [ ] Go to **Actions** → **Verify Package**
|
||||
- [ ] Click **Run workflow** with:
|
||||
- **source**: `pypi`
|
||||
- [ ] Wait for all tests to pass
|
||||
|
||||
### GitHub Release
|
||||
|
||||
- [ ] Create release from tag:
|
||||
```bash
|
||||
gh release create vX.Y.Z --title "vX.Y.Z" --notes "$(cat CHANGELOG.md | sed -n '/## \[X.Y.Z\]/,/## \[/p' | sed '$d')"
|
||||
```
|
||||
Or manually:
|
||||
- Go to **Releases** → **Draft a new release**
|
||||
- Select tag `vX.Y.Z`
|
||||
- Title: `vX.Y.Z`
|
||||
- Copy release notes from `CHANGELOG.md`
|
||||
- Publish release
|
||||
|
||||
- [ ] Publishing a **stable** release fires `publish-mcpb.yml`, which builds
|
||||
`notebooklm-mcp.mcpb` and attaches it to the release as an asset (it re-checks
|
||||
that the manifest, tag, and `pyproject.toml` versions all agree). Confirm the
|
||||
asset appears under the release once the workflow finishes — that is the
|
||||
one-click Claude Desktop bundle users download. Pre-releases skip this step by
|
||||
design (the thin launcher resolves the latest stable server, not the
|
||||
pre-release), so a `vX.Y.ZaN` release carries no `.mcpb`.
|
||||
|
||||
### Prune the API-Compat Allowlist
|
||||
|
||||
Pushing a **stable** tag advances the audit baseline — `audit_public_api_compat.py`
|
||||
resolves the latest reachable stable release tag (pre-releases are skipped), so
|
||||
the breaks you just shipped are now part of the `vX.Y.Z` baseline and are **no
|
||||
longer breaks against it**. (A pre-release tag such as `v0.8.0a1` does **not**
|
||||
advance the baseline, so the allowlist survives the whole pre-release cycle and
|
||||
prunes once, here, at the final `vX.Y.Z`.) The baseline advances automatically;
|
||||
the allowlist is the manual half that must reset, or it accumulates dead entries
|
||||
that describe nothing.
|
||||
|
||||
The lifecycle to keep in mind:
|
||||
|
||||
- **baseline** = the last *stable released* version (automatic — the latest
|
||||
stable release tag; pre-releases are skipped).
|
||||
- **allowlist** = the intentional breaks pending the *next* release. It should
|
||||
reset to (near) empty at each release boundary.
|
||||
|
||||
Concretely, **after the tag is pushed**, prune the entries that just shipped:
|
||||
|
||||
- [ ] In a follow-up PR (on `main`, after the tag exists), remove from
|
||||
`scripts/api-compat-allowlist.json` every `allowed_breaks` entry that
|
||||
described a `vPREV → vX.Y.Z` change. These are now baked into the `vX.Y.Z`
|
||||
baseline. List the stale entries with:
|
||||
```bash
|
||||
uv run python scripts/audit_public_api_compat.py --json \
|
||||
| python -c "import json,sys; print('\n'.join(f\"{e['code']} {e['object']}\" for e in json.load(sys.stdin)['stale_allowances']))"
|
||||
```
|
||||
- [ ] Re-run the gate in strict mode — it must report **no stale entries**:
|
||||
```bash
|
||||
uv run python scripts/audit_public_api_compat.py --check-stale
|
||||
```
|
||||
|
||||
> **Forcing function:** the Code Quality job runs the audit with `--check-stale`,
|
||||
> which **fails** on any allowlist entry that matches no break against the
|
||||
> baseline. So the moment `vX.Y.Z` is tagged, the just-shipped entries become
|
||||
> stale and CI goes red until this prune PR lands — the prune is mandatory, not
|
||||
> a checklist nicety. The pair-aware rule keeps the two path-views of a callable
|
||||
> (`notebooklm.X` and `notebooklm.client.X`) together: a unit is pruned only when
|
||||
> *neither* view still matches a break.
|
||||
|
||||
---
|
||||
|
||||
## Cleanup
|
||||
|
||||
### Remove Release Worktree
|
||||
|
||||
- [ ] Return to main repo:
|
||||
```bash
|
||||
cd /path/to/notebooklm-py
|
||||
```
|
||||
- [ ] Remove the release worktree:
|
||||
```bash
|
||||
git worktree remove .worktrees/vX.Y.Z
|
||||
```
|
||||
- [ ] Delete the local branch (if not already deleted by PR merge):
|
||||
```bash
|
||||
git branch -d release/vX.Y.Z
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### CI fails on PR
|
||||
|
||||
Fix issues in the release worktree and push again:
|
||||
```bash
|
||||
# In release worktree
|
||||
git add -A
|
||||
git commit -m "fix: address CI failures"
|
||||
git push
|
||||
```
|
||||
|
||||
### Need to abort release
|
||||
|
||||
```bash
|
||||
# Close the PR without merging
|
||||
gh pr close
|
||||
|
||||
# Remove worktree
|
||||
git worktree remove .worktrees/vX.Y.Z
|
||||
|
||||
# Delete local branch
|
||||
git branch -D release/vX.Y.Z
|
||||
|
||||
# Delete remote branch (if pushed)
|
||||
git push origin --delete release/vX.Y.Z
|
||||
```
|
||||
|
||||
### Tag already exists
|
||||
|
||||
```bash
|
||||
# Delete local tag
|
||||
git tag -d vX.Y.Z
|
||||
|
||||
# Delete remote tag (if pushed)
|
||||
git push origin :refs/tags/vX.Y.Z
|
||||
```
|
||||
|
||||
### TestPyPI upload fails
|
||||
|
||||
- Check if version already exists on TestPyPI
|
||||
- TestPyPI doesn't allow re-uploading same version
|
||||
- Bump to next patch version if needed (for a pre-release, bump the pre-release serial, e.g. `0.8.0a1 → 0.8.0a2`)
|
||||
|
||||
---
|
||||
|
||||
## Pre-releases (alpha / beta / rc)
|
||||
|
||||
Pre-releases let you stage a release for early adopters without affecting normal
|
||||
users. PEP 440 pre-release versions flow through the existing build, tag, publish,
|
||||
and verify machinery unchanged, and `pip install notebooklm-py` will not serve
|
||||
them to normal users. Follow the normal checklist above, with these differences:
|
||||
|
||||
1. **Version format is canonical PEP 440, byte-identical everywhere.** Use
|
||||
`X.Y.ZaN` / `X.Y.ZbN` / `X.Y.ZrcN` (e.g. `0.8.0a1`), tag `vX.Y.ZaN`. Do **not**
|
||||
use non-canonical forms like `0.8.0-alpha.1`, `0.8.0.rc1`, or `0.8.0RC1`:
|
||||
`publish.yml` tag-validation and `verify-package.yml` do raw string compares,
|
||||
and Verify Package compares against the *normalized* `importlib.metadata.version()`
|
||||
— a non-canonical spelling passes tag-match but fails Verify Package with a
|
||||
spurious "version mismatch".
|
||||
2. **Serial progression** within one cycle: advance the serial
|
||||
`a1 → a2 → … → b1 → … → rc1 → … → X.Y.0` (final). Do **not** bump the patch
|
||||
between pre-releases.
|
||||
3. **A pre-release *is* the final version's real surface.** A `X.Y.ZaN` tag must
|
||||
already contain every breaking flip for that version with deprecation shims
|
||||
removed. The `tests/_guardrails/test_v080_release_gate.py` version parser
|
||||
truncates the pre-release suffix, so the v0.8.0 breaking-flip release-gate
|
||||
fires at `0.8.0a1`. A "soft" alpha that still carries shims is not supported.
|
||||
4. **Re-lock after the bump.** After editing `pyproject.toml`'s version, run
|
||||
`uv sync` so `uv.lock`'s workspace-package version matches, else CI `--frozen`
|
||||
installs fail as out-of-date.
|
||||
5. **Normal users are unaffected.** `pip install notebooklm-py` skips pre-releases;
|
||||
only `pip install --pre` or an exact `==0.8.0a1` pin selects one.
|
||||
6. **Changelog.** Keep pre-release changes accumulating under `## [Unreleased]`
|
||||
(or a single in-progress `## [0.8.0]` heading). Do **not** cut a dated
|
||||
`[X.Y.ZaN]` section per pre-release.
|
||||
7. **Baseline / allowlist: no special handling.** `audit_public_api_compat.py`
|
||||
keeps the baseline on the last *stable* tag through the whole pre-release cycle
|
||||
(pre-release tags are skipped), so the allowlist behaves exactly as for a normal
|
||||
release — one prune after the final `X.Y.0` tag.
|
||||
8. **Gates still apply; none are optional.** The public-API audit,
|
||||
pre-commit/mypy/pytest, CI on PR, TestPyPI, and Verify Package all run
|
||||
unchanged. Verify Package reads the version from the checked-out ref, so
|
||||
dispatch it on the pre-release branch. Because Verify Package runs E2E, there is
|
||||
no "skip E2E for a pre-release" shortcut.
|
||||
9. **GitHub release must be flagged as a pre-release, and the notes must extract
|
||||
the aggregate heading** (not a per-pre-release one, which would yield empty
|
||||
notes):
|
||||
```bash
|
||||
gh release create v0.8.0a1 --prerelease --title "v0.8.0a1" \
|
||||
--notes "$(sed -n '/## \[0.8.0\]/,/## \[/p' CHANGELOG.md | sed '$d')"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Version Numbering
|
||||
|
||||
**IMPORTANT:** Read [stability.md](stability.md) before deciding version bump.
|
||||
|
||||
| Change Type | Bump | Example |
|
||||
|-------------|------|---------|
|
||||
| RPC method ID fixes | PATCH | 0.1.0 → 0.1.1 |
|
||||
| Bug fixes | PATCH | 0.1.1 → 0.1.2 |
|
||||
| Internal improvements (logging, auth UX, CI) | PATCH | 0.1.2 → 0.1.3 |
|
||||
| **New public API** (new classes, methods in `__all__`) | MINOR | 0.1.3 → 0.2.0 |
|
||||
| Breaking changes to public API | MAJOR | 0.2.0 → 1.0.0 |
|
||||
|
||||
**Key distinction:** "New features" means new **public API surface** (additions to `__all__` in `__init__.py`). Internal improvements, better error messages, logging enhancements, and UX improvements are PATCH releases.
|
||||
@@ -0,0 +1,529 @@
|
||||
# RPC Development Guide
|
||||
|
||||
**Status:** Active
|
||||
**Last Updated:** 2026-06-11
|
||||
|
||||
This guide covers everything about NotebookLM's RPC protocol: capturing calls, debugging issues, and implementing new methods.
|
||||
|
||||
See also: [Python API Reference](python-api.md)
|
||||
|
||||
---
|
||||
|
||||
## Protocol Overview
|
||||
|
||||
NotebookLM uses Google's `batchexecute` RPC protocol.
|
||||
|
||||
### Key Concepts
|
||||
|
||||
| Term | Description |
|
||||
|------|-------------|
|
||||
| **batchexecute** | Google's internal RPC endpoint |
|
||||
| **RPC ID** | 6-character identifier (e.g., `wXbhsf`, `s0tc2d`) |
|
||||
| **f.req** | URL-encoded JSON payload |
|
||||
| **at** | CSRF token (SNlM0e value) |
|
||||
| **Anti-XSSI** | `)]}'` prefix on responses |
|
||||
|
||||
### Protocol Flow
|
||||
|
||||
```
|
||||
1. Build request: [[[rpc_id, json_params, null, "generic"]]]
|
||||
2. Encode to f.req parameter
|
||||
3. POST to /_/LabsTailwindUi/data/batchexecute
|
||||
4. Strip )]}' prefix from response
|
||||
5. Parse chunked JSON, extract result
|
||||
```
|
||||
|
||||
### Source of Truth
|
||||
|
||||
- **RPC method IDs:** `src/notebooklm/rpc/types.py`
|
||||
- **Payload builders:** the owning implementation modules, for example
|
||||
`_notebooks.py::build_create_notebook_params`,
|
||||
`_source/upload_payloads.py`, `_source/add.py`, `_label/params.py`, and
|
||||
`_artifact/payloads.py`
|
||||
- **Golden payload tests:** `tests/unit/test_rpc_golden_payloads.py` and
|
||||
feature-specific unit tests such as `tests/unit/test_label_params.py`
|
||||
- **Human reference:** `docs/rpc-reference.md`, updated after the builder and
|
||||
tests land
|
||||
|
||||
---
|
||||
|
||||
## Capturing RPC Calls
|
||||
|
||||
### Manual Capture (Chrome DevTools)
|
||||
|
||||
Best for quick investigation and bug reports.
|
||||
|
||||
1. Open Chrome → Navigate to `https://notebooklm.google.com/`
|
||||
2. Open DevTools (`F12` or `Cmd+Option+I`)
|
||||
3. Go to **Network** tab
|
||||
4. Configure:
|
||||
- [x] **Preserve log**
|
||||
- [x] **Disable cache**
|
||||
5. Filter by: `batchexecute`
|
||||
6. **Perform ONE action** (isolate the exact RPC call)
|
||||
7. Click the request to inspect
|
||||
|
||||
**From the request:**
|
||||
- **Headers tab → URL `rpcids`**: The RPC method ID
|
||||
- **Payload tab → `f.req`**: URL-encoded payload
|
||||
- **Response tab**: Starts with `)]}'` prefix
|
||||
|
||||
### Decoding the Payload
|
||||
|
||||
**Browser console:**
|
||||
```javascript
|
||||
const encoded = "..."; // Paste f.req value
|
||||
const decoded = decodeURIComponent(encoded);
|
||||
const outer = JSON.parse(decoded);
|
||||
console.log("RPC ID:", outer[0][0][0]);
|
||||
console.log("Params:", JSON.parse(outer[0][0][1]));
|
||||
```
|
||||
|
||||
**Python:**
|
||||
```python
|
||||
import json
|
||||
from urllib.parse import unquote
|
||||
|
||||
def decode_f_req(encoded: str) -> dict:
|
||||
decoded = unquote(encoded)
|
||||
outer = json.loads(decoded)
|
||||
inner = outer[0][0]
|
||||
return {
|
||||
"rpc_id": inner[0],
|
||||
"params": json.loads(inner[1]) if inner[1] else None,
|
||||
}
|
||||
```
|
||||
|
||||
### Playwright Automation
|
||||
|
||||
Best for systematic capture and CI integration.
|
||||
|
||||
```python
|
||||
from playwright.async_api import async_playwright
|
||||
import json
|
||||
from urllib.parse import unquote, parse_qs
|
||||
|
||||
async def setup_capture_session():
|
||||
playwright = await async_playwright().start()
|
||||
browser = await playwright.chromium.launch_persistent_context(
|
||||
user_data_dir="./browser_state",
|
||||
headless=False,
|
||||
)
|
||||
page = browser.pages[0] if browser.pages else await browser.new_page()
|
||||
captured_rpcs = []
|
||||
|
||||
def handle_request(request):
|
||||
if "batchexecute" in request.url:
|
||||
post_data = request.post_data
|
||||
if post_data and "f.req" in post_data:
|
||||
params = parse_qs(post_data)
|
||||
f_req = params.get("f.req", [None])[0]
|
||||
if f_req:
|
||||
decoded = decode_f_req(f_req)
|
||||
captured_rpcs.append(decoded)
|
||||
|
||||
page.on("request", handle_request)
|
||||
return page, captured_rpcs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Debugging Issues
|
||||
|
||||
### Enable Debug Mode
|
||||
|
||||
```bash
|
||||
# See what RPC IDs the server returns
|
||||
NOTEBOOKLM_DEBUG_RPC=1 notebooklm <command>
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
DEBUG: Looking for RPC ID: Ljjv0c
|
||||
DEBUG: Found RPC IDs in response: ['Ljjv0c']
|
||||
```
|
||||
|
||||
If IDs don't match, the method ID has changed - report it in a GitHub issue.
|
||||
|
||||
### Common Scenarios
|
||||
|
||||
#### "Session Expired" Errors
|
||||
|
||||
```python
|
||||
# Check CSRF token
|
||||
print(client.auth.csrf_token)
|
||||
|
||||
# Refresh auth
|
||||
await client.refresh_auth()
|
||||
```
|
||||
|
||||
**Solution:** Re-run `notebooklm login`
|
||||
|
||||
#### RPC Method Returns None
|
||||
|
||||
**Causes:**
|
||||
- Rate limiting (Google returns empty result)
|
||||
- Wrong RPC method ID
|
||||
- Incorrect parameter structure
|
||||
|
||||
**Debug:**
|
||||
```python
|
||||
# decode_response is an internal RPC helper (notebooklm.rpc.* is internal per
|
||||
# docs/stability.md); import it from its defining module for contributor debugging.
|
||||
from notebooklm.rpc.decoder import decode_response
|
||||
|
||||
raw_response = await http_client.post(...)
|
||||
print("Raw:", raw_response.text[:500])
|
||||
|
||||
result = decode_response(raw_response.text, "METHOD_ID")
|
||||
print("Parsed:", result)
|
||||
```
|
||||
|
||||
#### Parameter Order Issues
|
||||
|
||||
RPC parameters are **position-sensitive**:
|
||||
|
||||
```python
|
||||
# WRONG - missing positional elements
|
||||
params = [value, notebook_id]
|
||||
|
||||
# RIGHT - all positions filled
|
||||
params = [value, notebook_id, None, None, settings]
|
||||
```
|
||||
|
||||
**Debug:** Compare your params with captured traffic byte-by-byte.
|
||||
|
||||
#### Nested List Depth
|
||||
|
||||
Source IDs have different nesting requirements:
|
||||
|
||||
```python
|
||||
# Single nesting (some methods)
|
||||
["source_id"]
|
||||
|
||||
# Double nesting
|
||||
[["source_id"]]
|
||||
|
||||
# Triple nesting (artifact generation)
|
||||
[[["source_id"]]]
|
||||
|
||||
# Quad nesting (get_source_guide)
|
||||
[[[["source_id"]]]]
|
||||
```
|
||||
|
||||
**Debug:** Capture working traffic and count brackets.
|
||||
|
||||
### Response Parsing
|
||||
|
||||
```python
|
||||
import json
|
||||
import re
|
||||
|
||||
def parse_response(text: str, rpc_id: str):
|
||||
"""Parse batchexecute response."""
|
||||
# Strip anti-XSSI prefix
|
||||
if text.startswith(")]}'"):
|
||||
text = re.sub(r"^\)\]\}'\r?\n", "", text)
|
||||
|
||||
# Find wrb.fr chunk for our RPC ID
|
||||
for line in text.split("\n"):
|
||||
try:
|
||||
chunk = json.loads(line)
|
||||
if chunk[0] == "wrb.fr" and chunk[1] == rpc_id:
|
||||
result = chunk[2]
|
||||
return json.loads(result) if isinstance(result, str) else result
|
||||
except (json.JSONDecodeError, IndexError):
|
||||
continue
|
||||
return None
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding New RPC Methods
|
||||
|
||||
### Workflow
|
||||
|
||||
```
|
||||
1. Capture → 2. Decode → 3. Implement → 4. Test → 5. Document
|
||||
```
|
||||
|
||||
### Step 1: Capture
|
||||
|
||||
Use Chrome DevTools or Playwright (see above).
|
||||
|
||||
**What to capture:**
|
||||
- RPC ID from URL `rpcids` parameter
|
||||
- Decoded `f.req` payload
|
||||
- Response structure
|
||||
|
||||
### Step 2: Decode
|
||||
|
||||
Document each position in the params array:
|
||||
|
||||
```python
|
||||
# Example: ADD_SOURCE for URL after the Gemini-3.5 wire-shape migration
|
||||
params = [
|
||||
[[None, None, [url], None, None, None, None, None, None, None, 1]],
|
||||
notebook_id,
|
||||
[2, None, None, [1, None, None, None, None, None, None, None, None, None, [1]]],
|
||||
]
|
||||
```
|
||||
|
||||
Key patterns:
|
||||
- **Nested source IDs:** Count brackets carefully
|
||||
- **Fixed flags:** Arrays like `[2]`, `[1]` that don't change
|
||||
- **Optional positions:** Often `None`
|
||||
|
||||
### Step 3: Implement
|
||||
|
||||
**Add RPC method ID** (`src/notebooklm/rpc/types.py`):
|
||||
```python
|
||||
class RPCMethod(str, Enum):
|
||||
NEW_METHOD = "AbCdEf" # 6-char ID from capture
|
||||
```
|
||||
|
||||
**Add client method** (appropriate `_*.py` file):
|
||||
```python
|
||||
async def new_method(self, notebook_id: str, param: str) -> SomeResult:
|
||||
"""Short description.
|
||||
|
||||
Args:
|
||||
notebook_id: The notebook ID.
|
||||
param: Description.
|
||||
|
||||
Returns:
|
||||
Description of return value.
|
||||
"""
|
||||
params = [
|
||||
param, # Position 0
|
||||
notebook_id, # Position 1
|
||||
[2], # Position 2: Fixed flag
|
||||
]
|
||||
|
||||
result = await self._rpc.rpc_call(
|
||||
RPCMethod.NEW_METHOD,
|
||||
params,
|
||||
source_path=f"/notebook/{notebook_id}",
|
||||
)
|
||||
|
||||
if result is None:
|
||||
return None
|
||||
return SomeResult.from_api_response(result)
|
||||
```
|
||||
|
||||
**Add dataclass if needed** (`src/notebooklm/types.py`):
|
||||
```python
|
||||
@dataclass
|
||||
class SomeResult:
|
||||
id: str
|
||||
title: str
|
||||
|
||||
@classmethod
|
||||
def from_api_response(cls, data: list[Any]) -> "SomeResult":
|
||||
return cls(id=data[0], title=data[1])
|
||||
```
|
||||
|
||||
### Step 4: Test
|
||||
|
||||
**Unit test** (`tests/unit/`):
|
||||
```python
|
||||
def test_encode_new_method():
|
||||
params = ["value", "notebook_id", [2]]
|
||||
result = encode_rpc_request(RPCMethod.NEW_METHOD, params)
|
||||
assert result[0][0][0] == "AbCdEf"
|
||||
```
|
||||
|
||||
**Unit test with a fake RPC executor** (`tests/unit/`):
|
||||
```python
|
||||
@pytest.mark.asyncio
|
||||
async def test_new_method(mock_client):
|
||||
mock_response = ["result_id", "Result Title"]
|
||||
with patch('notebooklm._rpc_executor.RpcExecutor.rpc_call', new_callable=AsyncMock) as mock:
|
||||
mock.return_value = mock_response
|
||||
result = await mock_client.some_api.new_method("nb_id", "param")
|
||||
assert result.id == "result_id"
|
||||
```
|
||||
|
||||
**VCR-backed integration test** (`tests/integration/`) or authenticated E2E
|
||||
test (`tests/e2e/`):
|
||||
```python
|
||||
@pytest.mark.e2e
|
||||
@pytest.mark.asyncio
|
||||
async def test_new_method_e2e(client, read_only_notebook_id):
|
||||
result = await client.some_api.new_method(read_only_notebook_id, "param")
|
||||
assert result is not None
|
||||
```
|
||||
|
||||
### Step 5: Document
|
||||
|
||||
Update `docs/rpc-reference.md`:
|
||||
|
||||
```markdown
|
||||
### NEW_METHOD (`AbCdEf`)
|
||||
|
||||
**Purpose:** Short description
|
||||
|
||||
**Params:**
|
||||
```python
|
||||
params = [
|
||||
some_value, # 0: Description
|
||||
notebook_id, # 1: Notebook ID
|
||||
[2], # 2: Fixed flag
|
||||
]
|
||||
```
|
||||
|
||||
**Response:** Description of response structure
|
||||
|
||||
**Source:** `_some_api.py:123`
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Wrong nesting level
|
||||
|
||||
Different methods need different source ID nesting. Check similar methods.
|
||||
|
||||
### Position sensitivity
|
||||
|
||||
Params are arrays, not dicts. Position matters:
|
||||
|
||||
```python
|
||||
# WRONG - missing position 2
|
||||
params = [value, notebook_id, settings]
|
||||
|
||||
# RIGHT - explicit None for unused positions
|
||||
params = [value, notebook_id, None, settings]
|
||||
```
|
||||
|
||||
### Forgetting source_path
|
||||
|
||||
Some methods require `source_path` for routing:
|
||||
|
||||
```python
|
||||
# May fail without source_path
|
||||
await self._rpc.rpc_call(RPCMethod.X, params)
|
||||
|
||||
# Correct
|
||||
await self._rpc.rpc_call(
|
||||
RPCMethod.X,
|
||||
params,
|
||||
source_path=f"/notebook/{notebook_id}",
|
||||
)
|
||||
```
|
||||
|
||||
### Response parsing
|
||||
|
||||
API returns nested arrays. Print raw response first:
|
||||
|
||||
```python
|
||||
result = await self._rpc.rpc_call(...)
|
||||
print(f"DEBUG: {result}") # See actual structure
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] Captured RPC ID and params structure
|
||||
- [ ] Added to `RPCMethod` enum in `rpc/types.py`
|
||||
- [ ] Implemented method in appropriate `_*.py` file
|
||||
- [ ] Added dataclass if needed in `types.py`
|
||||
- [ ] Added CLI command if needed
|
||||
- [ ] Unit test for encoding
|
||||
- [ ] Integration test with mock
|
||||
- [ ] E2E test (manual verification OK for rare operations)
|
||||
- [ ] Updated `rpc-reference.md`
|
||||
|
||||
---
|
||||
|
||||
## LLM Agent Workflow
|
||||
|
||||
For AI agents discovering new RPC methods:
|
||||
|
||||
### Context
|
||||
|
||||
```
|
||||
NotebookLM Protocol Facts:
|
||||
- Endpoint: /_/LabsTailwindUi/data/batchexecute
|
||||
- RPC IDs are 6-character strings (e.g., "wXbhsf")
|
||||
- Payload: [[[rpc_id, json_params, null, "generic"]]]
|
||||
- Response has )]}' anti-XSSI prefix
|
||||
- Parameters are position-sensitive arrays
|
||||
|
||||
Source of Truth:
|
||||
- Canonical RPC IDs: src/notebooklm/rpc/types.py
|
||||
- Payload structures: docs/rpc-reference.md
|
||||
```
|
||||
|
||||
### Discovery Prompt Template
|
||||
|
||||
```
|
||||
Task: Discover the RPC call for [ACTION_NAME]
|
||||
|
||||
Steps:
|
||||
1. Identify the UI element that triggers this action
|
||||
2. Set up network interception for batchexecute
|
||||
3. Trigger the UI action
|
||||
4. Capture the RPC request
|
||||
|
||||
Document:
|
||||
- RPC ID (6-character string)
|
||||
- Payload structure with parameter positions
|
||||
- Source ID nesting pattern
|
||||
- Response structure
|
||||
```
|
||||
|
||||
### Validation
|
||||
|
||||
```python
|
||||
async def validate_root_rpc_call(method_name: str, params: list):
|
||||
from notebooklm import NotebookLMClient
|
||||
from notebooklm.rpc import RPCMethod
|
||||
|
||||
async with NotebookLMClient.from_storage() as client:
|
||||
# Public raw calls use the default root source path. For notebook-scoped
|
||||
# calls that need source_path="/notebook/<id>", prefer the typed
|
||||
# namespace API or a focused internal test around RpcExecutor.
|
||||
result = await client.rpc_call(RPCMethod[method_name], params)
|
||||
|
||||
assert result is not None, f"RPC {method_name} returned None"
|
||||
return {"method": method_name, "status": "verified"}
|
||||
```
|
||||
|
||||
## RPC Health Check Triage Policy
|
||||
|
||||
The `rpc-health.yml` workflow runs daily for `main` (07:00 UTC). Release branch
|
||||
health checks are manual via `custom_branch=release/vX.Y.Z`. The workflow opens
|
||||
an issue on any detected RPC ID mismatch, auth failure, or non-transient RPC
|
||||
error:
|
||||
|
||||
- **RPC ID mismatch** issues (exit code 1): labeled `bug, rpc-breakage, automated`.
|
||||
- **Auth failure** issues (exit code 2): labeled `bug, automated` (no `rpc-breakage`
|
||||
label — auth is an operational concern, not a protocol break).
|
||||
- **Non-transient ERROR detected** issues (exit code 3): labeled `rpc-error, bug,
|
||||
automated`. Opened when `check_rpc_health.py` surfaces failures that survive
|
||||
the rate-limit / `RESOURCE_EXHAUSTED` filter (timeouts, parse failures,
|
||||
unexpected HTTP errors). The issue body lists the affected method IDs
|
||||
extracted from the report, so triage can start without re-running the check.
|
||||
See the `Extract failing methods for ERROR issue` step in
|
||||
`.github/workflows/rpc-health.yml` for the body-assembly logic.
|
||||
|
||||
Routing:
|
||||
|
||||
- **Maintainer assignment**: Issues land in the `teng-lin/notebooklm-py`
|
||||
default issue inbox. The maintainer triages within 24 hours during business
|
||||
days. (No auto-assignee — the project has a single maintainer and
|
||||
auto-assignment adds noise.)
|
||||
- **Acknowledged-but-deferred**: If an upstream RPC change is observed but
|
||||
the library still functions for the majority of users (e.g., one optional
|
||||
field renamed), the maintainer closes the issue with the `acknowledged`
|
||||
label and links the PR that resolves it.
|
||||
- **Notifying users**: If the breakage affects an RPC most users invoke
|
||||
(e.g., `LIST_NOTEBOOKS`, `CREATE_NOTEBOOK`), the maintainer additionally
|
||||
files a release-note draft + pins the issue.
|
||||
|
||||
If you see an `rpc-breakage` issue sitting unattended for >7 days, ping the
|
||||
maintainer in a comment — it likely fell out of the inbox. The intent of this
|
||||
workflow is fast detection, not perpetual auto-noise.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,453 @@
|
||||
# API Stability and Versioning
|
||||
|
||||
**Status:** Active
|
||||
**Last Updated:** 2026-07-04
|
||||
|
||||
This document describes the stability guarantees and versioning policy for `notebooklm-py`.
|
||||
|
||||
## Important Context
|
||||
|
||||
This library uses **undocumented Google APIs**. Unlike official Google APIs, there are:
|
||||
|
||||
- **No stability guarantees from Google**
|
||||
- **No deprecation notices** before changes
|
||||
- **No SLAs or support**
|
||||
|
||||
Google can change the underlying APIs at any time, which may break this library without warning.
|
||||
|
||||
## Versioning Policy
|
||||
|
||||
We follow [Semantic Versioning](https://semver.org/) with modifications for our unique situation:
|
||||
|
||||
### Version Format: `MAJOR.MINOR.PATCH`
|
||||
|
||||
| Change Type | Version Bump | Example |
|
||||
|-------------|--------------|---------|
|
||||
| RPC method ID fixes (Google changed something) | PATCH | 0.1.0 → 0.1.1 |
|
||||
| Bug fixes | PATCH | 0.1.1 → 0.1.2 |
|
||||
| New features (backward compatible) | MINOR | 0.1.2 → 0.2.0 |
|
||||
| Public API breaking changes | MAJOR | 0.2.0 → 1.0.0 |
|
||||
|
||||
### Special Considerations
|
||||
|
||||
1. **RPC ID Changes = Patch Release**
|
||||
- When Google changes internal RPC method IDs, we release a patch
|
||||
- These are "bug fixes" from our perspective, not breaking changes
|
||||
- Users should always use the latest patch version
|
||||
|
||||
2. **Python API Stability**
|
||||
- Public API (items in `__all__`) is stable within a major version
|
||||
- Breaking changes require a major version bump
|
||||
- Deprecated APIs are marked with `DeprecationWarning` and documented
|
||||
|
||||
3. **0.x Pre-1.0 Semantics**
|
||||
- Per [SemVer §4](https://semver.org/#spec-item-4), the project is currently in 0.x and the public API is not yet considered stable.
|
||||
- **MINOR** releases (e.g. 0.4.0 → 0.5.0) **may remove** previously deprecated public APIs. Removal is preceded by at least one MINOR release of `DeprecationWarning` notice.
|
||||
- Once the project reaches 1.0.0, breaking changes will require a **MAJOR** bump as described above.
|
||||
|
||||
## Public API Surface
|
||||
|
||||
The following are considered **public API** and are subject to stability guarantees:
|
||||
|
||||
### Stable (Won't break without major version bump)
|
||||
|
||||
```python
|
||||
# Version
|
||||
__version__ # Package version string (read-only)
|
||||
|
||||
# Client
|
||||
NotebookLMClient
|
||||
NotebookLMClient.from_storage()
|
||||
NotebookLMClient.notebooks
|
||||
NotebookLMClient.sources
|
||||
NotebookLMClient.artifacts
|
||||
NotebookLMClient.chat
|
||||
NotebookLMClient.research
|
||||
NotebookLMClient.notes
|
||||
NotebookLMClient.settings
|
||||
NotebookLMClient.sharing
|
||||
NotebookLMClient.labels
|
||||
NotebookLMClient.mind_maps
|
||||
NotebookLMClient.rpc_call()
|
||||
|
||||
# Types
|
||||
Notebook, Source, Artifact, Note, Label, MindMap
|
||||
GenerationState, GenerationStatus, AskResult
|
||||
NotebookDescription, ConversationTurn
|
||||
ShareStatus, SharedUser, SourceFulltext
|
||||
NotebookMetadata, SourceSummary
|
||||
AccountLimits
|
||||
ChatReference, ReportSuggestion, PromptSuggestion, SuggestedTopic
|
||||
MindMapKind, MindMapResult
|
||||
ResearchStart, ResearchStatus, ResearchTask, ResearchSource
|
||||
ClientMetricsSnapshot, ConnectionLimits, RpcTelemetryEvent
|
||||
|
||||
# Exceptions (all inherit from NotebookLMError)
|
||||
NotebookLMError # Base exception
|
||||
NotFoundError # Cross-domain umbrella for *NotFoundError
|
||||
WaitTimeoutError # Cross-domain umbrella for wait/poll timeouts (also a built-in TimeoutError)
|
||||
RPCError, AuthError, RateLimitError, RPCTimeoutError, ServerError
|
||||
NetworkError, DecodingError, UnknownRPCMethodError
|
||||
ClientError, ConfigurationError, ValidationError
|
||||
# Domain-specific
|
||||
# Note: *NotFoundError classes mix in RPCError (catchable as either RPCError
|
||||
# or the domain base). v0.6.0 restored this symmetry across all three "not
|
||||
# found" types — see docs/python-api.md#error-handling for migration prose.
|
||||
# Note: *TimeoutError classes mix in WaitTimeoutError (and the built-in
|
||||
# TimeoutError). v0.7.0 added the WaitTimeoutError umbrella so `except
|
||||
# WaitTimeoutError` catches source/artifact/research wait timeouts uniformly,
|
||||
# while `except TimeoutError` keeps working — see docs/python-api.md#waittimeouterror.
|
||||
SourceError, SourceAddError, SourceProcessingError, SourceTimeoutError, SourceNotFoundError
|
||||
NotebookError, NotebookNotFoundError
|
||||
ArtifactError, ArtifactDownloadError, ArtifactFeatureUnavailableError, ArtifactNotFoundError, ArtifactNotReadyError, ArtifactParseError
|
||||
ArtifactTimeoutError, ArtifactPendingTimeoutError, ArtifactInProgressTimeoutError
|
||||
ResearchError, ResearchTimeoutError, ResearchTaskMismatchError, AmbiguousResearchTaskError
|
||||
# Note: notes.get/update/delete and mind_maps.get/rename/delete now raise
|
||||
# their domain *NotFoundError on a missing target; use get_or_none() for
|
||||
# warning-free None-on-miss lookups.
|
||||
NoteError, NoteNotFoundError
|
||||
MindMapError, MindMapNotFoundError
|
||||
LabelError, LabelNotFoundError
|
||||
ChatError, ChatResponseParseError
|
||||
|
||||
# Enums
|
||||
AudioFormat, AudioLength
|
||||
VideoFormat, VideoStyle
|
||||
QuizQuantity, QuizDifficulty
|
||||
InfographicOrientation, InfographicDetail, InfographicStyle
|
||||
SlideDeckFormat, SlideDeckLength
|
||||
ReportFormat
|
||||
SourceType, ArtifactType, SourceStatus
|
||||
ShareAccess, SharePermission, ShareViewLevel
|
||||
ChatGoal, ChatResponseLength, ChatMode
|
||||
DriveMimeType, ExportType
|
||||
|
||||
# Auth
|
||||
AuthTokens
|
||||
notebooklm.paths.get_storage_path()
|
||||
|
||||
# Logging and Correlation
|
||||
notebooklm.configure_logging
|
||||
notebooklm.get_request_id
|
||||
notebooklm.set_request_id
|
||||
notebooklm.reset_request_id
|
||||
notebooklm.correlation_id
|
||||
|
||||
# Citation and Research Helpers
|
||||
notebooklm.utils.resolve_chat_reference_passage
|
||||
notebooklm.research.select_cited_sources
|
||||
notebooklm.research.normalize_url
|
||||
notebooklm.research.extract_report_urls
|
||||
|
||||
# Helpers (cookies extra) - imported from notebooklm.auth
|
||||
notebooklm.auth.convert_rookiepy_cookies_to_storage_state # requires `pip install "notebooklm-py[cookies]"` — see docs/installation.md#optional-extras-matrix
|
||||
|
||||
# Cookie-domain tiers - imported from notebooklm.auth
|
||||
notebooklm.auth.REQUIRED_COOKIE_DOMAINS
|
||||
notebooklm.auth.OPTIONAL_COOKIE_DOMAINS
|
||||
notebooklm.auth.OPTIONAL_COOKIE_DOMAINS_BY_LABEL
|
||||
```
|
||||
|
||||
### Internal helpers exported for compatibility
|
||||
|
||||
The following symbols appear in `notebooklm/__all__` so that downstream code can
|
||||
import them via `from notebooklm import ...`, but they are **not** covered by
|
||||
the stability guarantee above. They support narrow integration use cases
|
||||
(typed exception handling and warning filters) and may be
|
||||
renamed, narrowed, or removed in a future minor release. Prefer the stable
|
||||
surface when possible.
|
||||
|
||||
```python
|
||||
CitedSourceSelection # Chat citation payload — internal shape, exposed for typing
|
||||
AuthExtractionError # Specialized AuthError raised by browser-based login
|
||||
NotebookLimitError # Raised when account notebook quota is exhausted
|
||||
UnknownTypeWarning # Warning category emitted when .kind falls back to UNKNOWN
|
||||
```
|
||||
|
||||
### Internal (May change without notice)
|
||||
|
||||
```python
|
||||
# These are NOT part of the public API:
|
||||
notebooklm.rpc.* # RPC protocol internals, except documented power-user imports
|
||||
notebooklm._*.py # All underscore-prefixed modules
|
||||
notebooklm.auth.* # Auth internals (except documented AuthTokens, cookie conversion, and cookie-domain constants)
|
||||
```
|
||||
|
||||
For raw-RPC power-user calls, import the documented RPC helpers explicitly:
|
||||
```python
|
||||
from notebooklm.rpc import RPCMethod, resolve_rpc_id
|
||||
```
|
||||
|
||||
### Adapter surfaces: MCP server and REST API (experimental)
|
||||
|
||||
The **MCP tool surface** (`mcp` extra, `notebooklm-mcp`) and the **single-tenant
|
||||
REST API** (`server` extra, `notebooklm-server`) are transport adapters over the
|
||||
same `_app/` business logic as the CLI. They are **experimental and not covered
|
||||
by the semver guarantees above** — tool/route names, parameters, and response
|
||||
shapes may change between releases without a major-version bump. The underlying
|
||||
Python client API they call is still governed by the stability policy; only the
|
||||
adapter surfaces are exempt. The remote-MCP connector (HTTP transport,
|
||||
self-hosted OAuth, Docker/Cloudflare/Tailscale deployment) is likewise
|
||||
experimental.
|
||||
|
||||
### Strict decoding (the only mode since v0.7.0)
|
||||
|
||||
Schema-drift helpers (notably the internal ``safe_index`` decode helper) **raise**
|
||||
:class:`~notebooklm.exceptions.UnknownRPCMethodError` when Google's
|
||||
batchexecute response shape does not match what the decoder expects. This is
|
||||
now the only behavior: the legacy ``NOTEBOOKLM_STRICT_DECODE=0`` warn-and-
|
||||
return-``None`` opt-out was retired in v0.7.0 (it had a one-release
|
||||
``DeprecationWarning`` window through v0.5.0/v0.6.0). The env var is now
|
||||
ignored.
|
||||
|
||||
Stability implications:
|
||||
|
||||
- **Exception type is stable.** ``UnknownRPCMethodError`` is a subclass of
|
||||
``DecodingError`` and ``RPCError`` (both public-API exceptions). Code that
|
||||
already catches ``RPCError`` continues to handle drift correctly.
|
||||
- **No silent shape changes.** Methods that previously returned ``None`` /
|
||||
empty values on drift now raise. Callers that treated ``None`` as a valid
|
||||
sentinel must add an ``except UnknownRPCMethodError`` branch.
|
||||
|
||||
See [`docs/configuration.md#decoder-strictness`](configuration.md#decoder-strictness)
|
||||
for the env-var contract and
|
||||
[`docs/adr/0011-schema-validation-policy.md`](adr/0011-schema-validation-policy.md)
|
||||
for the design rationale behind the strict-decode policy.
|
||||
|
||||
## Deprecation Policy
|
||||
|
||||
1. **Deprecation Notice**: Deprecated features emit `DeprecationWarning`
|
||||
2. **Documentation**: Deprecations are noted in docstrings and CHANGELOG
|
||||
3. **Removal Timeline**: Deprecated features are removed in the next major version. While the project is in 0.x, removal may instead occur in the next MINOR release after at least one MINOR cycle of `DeprecationWarning` (see "0.x Pre-1.0 Semantics" above).
|
||||
4. **Migration Guide**: Breaking changes include migration instructions
|
||||
|
||||
### Currently Deprecated
|
||||
|
||||
See [`docs/deprecations.md`](deprecations.md) for the canonical list of
|
||||
currently-deprecated APIs and their scheduled removal versions, plus the
|
||||
deprecations removed in v0.6.0, v0.7.0, and v0.8.0.
|
||||
|
||||
### Removed in v0.5.0
|
||||
|
||||
The following v0.3-era deprecations completed their removal cycle in v0.5.0:
|
||||
|
||||
| Removed | Replacement | Notes |
|
||||
|---------|-------------|-------|
|
||||
| `Source.source_type` | `Source.kind` | Returns `SourceType` str enum |
|
||||
| `Artifact.artifact_type` | `Artifact.kind` | Returns `ArtifactType` str enum |
|
||||
| `Artifact.variant` | `Artifact.kind` | Use `.is_quiz` / `.is_flashcards` |
|
||||
| `SourceFulltext.source_type` | `SourceFulltext.kind` | Returns `SourceType` str enum |
|
||||
| `notebooklm.StudioContentType` | `ArtifactType` | Str enum for user-facing code |
|
||||
| `notebooklm.DEFAULT_STORAGE_PATH` | `notebooklm.paths.get_storage_path()` | Module-level constant replaced by helper |
|
||||
| `notebooklm.rpc.types.StudioContentType` | `ArtifactType` | Internal raw code alias removed |
|
||||
| `notebooklm.rpc.StudioContentType` | `ArtifactType` | Internal re-export removed |
|
||||
| `notebooklm.rpc.RPCMethod.DISCOVER_SOURCES` | none | Unused raw RPC enum member, not exercised by client APIs |
|
||||
| `notebooklm.rpc.RPCMethod.QUERY_ENDPOINT` | `notebooklm.rpc.get_query_url()` (internal) | Endpoint URL path moved out of the RPC method enum; `get_query_url()` is itself internal plumbing (`notebooklm.rpc.*` is internal — see above) with no blessed public replacement |
|
||||
| `notebooklm.cli.language_cmd.save_config` | `_save_config` | Private low-level write primitive only |
|
||||
|
||||
### Deprecated for a future major release
|
||||
|
||||
| Deprecated | Replacement | Notes |
|
||||
|------------|-------------|-------|
|
||||
| Awaiting `NotebookLMClient.from_storage(...)` | `async with NotebookLMClient.from_storage(...) as client:` | Emits `DeprecationWarning`; scheduled for v1.0 removal |
|
||||
|
||||
### Permanent aliases
|
||||
|
||||
`RPCError.rpc_id` and `RPCError.code` are permanent backward-compatibility
|
||||
aliases for `RPCError.method_id` and `RPCError.rpc_code`. Exception diagnostic
|
||||
aliases are exempt from the standard deprecation cycle because removal can mask
|
||||
the original exception inside `except` handlers. New code should prefer the
|
||||
canonical attribute names, but existing exception handlers may keep using the
|
||||
aliases.
|
||||
|
||||
## Migration Guides
|
||||
|
||||
### Migrating from v0.3.x to v0.4.0
|
||||
|
||||
Version 0.4.0 is backward compatible with v0.3.x. Notable additions:
|
||||
|
||||
- **Multi-account profiles** - Existing single-account setups continue to work as the implicit default profile. Your existing `~/.notebooklm/storage_state.json` is auto-detected — no manual migration is required. New accounts can be added via `notebooklm profile create <name>`.
|
||||
- **`[cookies]` optional extra** - To reuse cookies from your existing browser, install with `pip install "notebooklm-py[cookies]"` (requires `rookiepy`; full extras matrix: [docs/installation.md#optional-extras-matrix](installation.md#optional-extras-matrix)).
|
||||
- **Deprecation removal deferred** - The deprecated attributes originally scheduled for v0.4.0 (`Source.source_type`, `Artifact.artifact_type`, `Artifact.variant`, `SourceFulltext.source_type`, `StudioContentType`, `DEFAULT_STORAGE_PATH`) were deferred to v0.5.0. In v0.5.0 and later, use the replacements listed in [Removed in v0.5.0](#removed-in-v050).
|
||||
|
||||
### Migrating from v0.2.x to v0.3.0
|
||||
|
||||
Version 0.3.0 introduced attributes that were deprecated until their v0.5.0
|
||||
removal. The historical migration examples below show the replacement surface.
|
||||
|
||||
#### 1. `Source.source_type` → `Source.kind`
|
||||
|
||||
**Before (removed in v0.5.0):**
|
||||
```python
|
||||
source = (await client.sources.list(notebook_id))[0]
|
||||
if source.source_type == "pdf":
|
||||
print("This is a PDF")
|
||||
```
|
||||
|
||||
**After (recommended):**
|
||||
```python
|
||||
from notebooklm import SourceType
|
||||
|
||||
source = (await client.sources.list(notebook_id))[0]
|
||||
|
||||
# Option 1: Use enum comparison (recommended)
|
||||
if source.kind == SourceType.PDF:
|
||||
print("This is a PDF")
|
||||
|
||||
# Option 2: Use string comparison (str enum supports this)
|
||||
if source.kind == "pdf":
|
||||
print("This is a PDF")
|
||||
```
|
||||
|
||||
**Available `SourceType` values:**
|
||||
`GOOGLE_DOCS`, `GOOGLE_SLIDES`, `GOOGLE_SPREADSHEET`, `PDF`, `PASTED_TEXT`, `WEB_PAGE`, `YOUTUBE`, `MARKDOWN`, `DOCX`, `CSV`, `IMAGE`, `MEDIA`, `UNKNOWN`
|
||||
|
||||
#### 2. `Artifact.artifact_type` → `Artifact.kind`
|
||||
|
||||
**Before (removed in v0.5.0):**
|
||||
```python
|
||||
artifact = (await client.artifacts.list(notebook_id))[0]
|
||||
if artifact.artifact_type == 1:
|
||||
print("This is an audio artifact")
|
||||
```
|
||||
|
||||
**After (recommended):**
|
||||
```python
|
||||
from notebooklm import ArtifactType
|
||||
|
||||
artifact = (await client.artifacts.list(notebook_id))[0]
|
||||
|
||||
# Option 1: Use enum comparison (recommended)
|
||||
if artifact.kind == ArtifactType.AUDIO:
|
||||
print("This is an audio artifact")
|
||||
|
||||
# Option 2: Use string comparison (str enum supports this)
|
||||
if artifact.kind == "audio":
|
||||
print("This is an audio artifact")
|
||||
```
|
||||
|
||||
**Available `ArtifactType` values:**
|
||||
`AUDIO`, `VIDEO`, `REPORT`, `QUIZ`, `FLASHCARDS`, `MIND_MAP`, `INFOGRAPHIC`, `SLIDE_DECK`, `DATA_TABLE`, `UNKNOWN`
|
||||
|
||||
#### 3. `Artifact.variant` → `Artifact.kind` or helpers
|
||||
|
||||
**Before (removed in v0.5.0):**
|
||||
```python
|
||||
if artifact.artifact_type == 4 and artifact.variant == 2:
|
||||
print("This is a quiz")
|
||||
```
|
||||
|
||||
**After (recommended):**
|
||||
```python
|
||||
# Option 1: Use .kind
|
||||
if artifact.kind == ArtifactType.QUIZ:
|
||||
print("This is a quiz")
|
||||
|
||||
# Option 2: Use helper properties
|
||||
if artifact.is_quiz:
|
||||
print("This is a quiz")
|
||||
if artifact.is_flashcards:
|
||||
print("These are flashcards")
|
||||
```
|
||||
|
||||
#### Why These Changes?
|
||||
|
||||
1. **Stability**: The `.kind` property abstracts internal integer codes that Google may change
|
||||
2. **Usability**: String enums work in comparisons, logging, and serialization
|
||||
3. **Future-proofing**: Unknown types return `UNKNOWN` with a warning instead of crashing
|
||||
|
||||
## What Happens When Google Breaks Things
|
||||
|
||||
When Google changes their internal APIs:
|
||||
|
||||
1. **Detection**: Automated RPC health check runs nightly for `main`; release
|
||||
branch checks run manually during release prep (see below)
|
||||
2. **Investigation**: Identify changed method IDs using browser devtools
|
||||
3. **Fix**: Update `rpc/types.py` with new method IDs
|
||||
4. **Release**: Push patch release as soon as possible
|
||||
|
||||
### Automated RPC Health Check
|
||||
|
||||
A nightly GitHub Action (`rpc-health.yml`) monitors all 35+ RPC methods for ID
|
||||
changes on `main`. Release branches use the same workflow through manual
|
||||
dispatch.
|
||||
|
||||
**What it verifies:**
|
||||
- The RPC method ID we send matches the ID returned in the response envelope
|
||||
- Example: `LIST_NOTEBOOKS` sends `wXbhsf` → response must contain `wXbhsf`
|
||||
|
||||
**What it does NOT verify:**
|
||||
- Response data correctness (E2E tests cover this)
|
||||
- Response schema validation (too fragile across 35+ methods)
|
||||
- Business logic (out of scope for monitoring)
|
||||
|
||||
**Why this design:**
|
||||
- Google's breaking change pattern is silent ID changes, not schema changes
|
||||
- Error responses still contain the method ID, so we detect mismatches even on API errors
|
||||
- A mismatch means `rpc/types.py` needs updating, triggering a patch release
|
||||
|
||||
**On mismatch detection:**
|
||||
- GitHub Issue auto-created with `bug`, `rpc-breakage`, and `automated` labels
|
||||
- Report shows expected vs actual IDs and which `RPCMethod` entries need updating
|
||||
|
||||
**Manual trigger:** `gh workflow run rpc-health.yml -f custom_branch=release/vX.Y.Z`
|
||||
|
||||
### How to Report API Breakage
|
||||
|
||||
1. Check [GitHub Issues](https://github.com/teng-lin/notebooklm-py/issues) for existing reports
|
||||
2. If not reported, open an issue with:
|
||||
- Error message (especially any RPC error codes)
|
||||
- Which operation failed
|
||||
- When it started failing
|
||||
3. See [RPC Development Guide](rpc-development.md) for debugging
|
||||
|
||||
### Self-Recovery
|
||||
|
||||
If the library breaks before we release a fix:
|
||||
|
||||
1. Open browser devtools on NotebookLM
|
||||
2. Perform the failing operation manually
|
||||
3. Find the new RPC method ID in Network tab
|
||||
4. Temporarily override the rotated ID without mutating the enum:
|
||||
```bash
|
||||
export NOTEBOOKLM_RPC_OVERRIDES='{"LIST_NOTEBOOKS": "NewMethodId"}'
|
||||
```
|
||||
|
||||
The override key is the `RPCMethod` enum name. See
|
||||
[configuration.md#environment-variables](configuration.md#environment-variables)
|
||||
for validation and host-allowlist behavior.
|
||||
|
||||
## Upgrade Recommendations
|
||||
|
||||
### Stay Current
|
||||
|
||||
```bash
|
||||
# Always use latest patch version
|
||||
pip install --upgrade notebooklm-py
|
||||
```
|
||||
|
||||
### Pin Appropriately
|
||||
|
||||
```toml
|
||||
# pyproject.toml - recommended
|
||||
dependencies = [
|
||||
"notebooklm-py>=0.1,<1.0", # Accept patches and minors
|
||||
]
|
||||
|
||||
# requirements.txt - for reproducibility
|
||||
notebooklm-py==0.1.0 # Exact version (but update regularly!)
|
||||
```
|
||||
|
||||
### Test Before Upgrading
|
||||
|
||||
```bash
|
||||
# Test in development first
|
||||
pip install notebooklm-py==X.Y.Z
|
||||
pytest
|
||||
```
|
||||
|
||||
## Questions?
|
||||
|
||||
- **Bug reports**: [GitHub Issues](https://github.com/teng-lin/notebooklm-py/issues)
|
||||
- **Discussions**: [GitHub Discussions](https://github.com/teng-lin/notebooklm-py/discussions)
|
||||
- **Security issues**: See [SECURITY.md](../SECURITY.md)
|
||||
@@ -0,0 +1,837 @@
|
||||
# Troubleshooting
|
||||
|
||||
**Status:** Active
|
||||
**Last Updated:** 2026-07-04
|
||||
|
||||
Common issues, known limitations, and workarounds for `notebooklm-py`.
|
||||
|
||||
## Common Errors
|
||||
|
||||
### Authentication Errors
|
||||
|
||||
**First step:** Run `notebooklm auth check` to diagnose auth issues:
|
||||
```bash
|
||||
notebooklm auth check # Quick local validation
|
||||
notebooklm auth check --test # Full validation with network test
|
||||
notebooklm auth check --json # Machine-readable output for CI/CD
|
||||
```
|
||||
|
||||
This shows:
|
||||
- Storage file location and validity
|
||||
- Which cookies are present and their domains
|
||||
- Whether NOTEBOOKLM_AUTH_JSON or NOTEBOOKLM_HOME is being used
|
||||
- (With `--test`) Whether token fetch succeeds
|
||||
|
||||
#### Automatic Token Refresh
|
||||
|
||||
The client **automatically refreshes** CSRF tokens when authentication errors are detected. This happens transparently:
|
||||
|
||||
- When an RPC call fails with an auth error, the client:
|
||||
1. Fetches fresh CSRF token and session ID from the NotebookLM homepage
|
||||
2. Waits briefly to avoid rate limiting
|
||||
3. Retries the failed request once
|
||||
- Concurrent requests share a single refresh task to prevent token thrashing
|
||||
- If refresh fails, the original error is raised with the refresh failure as cause
|
||||
|
||||
This means most "CSRF token expired" errors resolve automatically.
|
||||
|
||||
#### Cookie freshness for long-running / unattended use
|
||||
|
||||
Google rotates `__Secure-1PSIDTS` (the freshness partner of `__Secure-1PSID`) on its own cadence; the on-disk `Expires` field is **not** a reliable predictor of server-side validity. The library handles freshness in layered fallbacks, ordered cheapest to heaviest:
|
||||
|
||||
1. **Per-call rotation poke** (default ON) — every `fetch_tokens` makes a best-effort POST to `accounts.google.com/RotateCookies`. Disable with `NOTEBOOKLM_DISABLE_KEEPALIVE_POKE=1`.
|
||||
2. **Periodic background poke** — pass `keepalive=<seconds>` to `NotebookLMClient` for clients held open for hours.
|
||||
3. **Layer-3 headless re-auth** — explicit Python opt-in via `await client.refresh_auth(allow_headless=True)`, or automatic mid-RPC opt-in with `NOTEBOOKLM_HEADLESS_REAUTH=1`. This drives the persisted browser profile, or attaches to a loopback Chrome DevTools endpoint from `NOTEBOOKLM_HEADLESS_REAUTH_CDP_URL`. Treat CDP as account-equivalent: only use `127.0.0.1` / `localhost`, never a remote browser.
|
||||
4. **Layer-4 master-token re-mint** — when a `master_token.json` sits beside the profile's `storage_state.json` (the `[headless]` extra; `notebooklm login --master-token`), a fully-expired session re-mints fresh cookies from the durable master token in-process, after layers 1–3 are exhausted. This is what makes long-lived headless workers self-heal without a browser. See [installation.md#d-headless-server-or-ci](installation.md#d-headless-server-or-ci).
|
||||
5. **External recovery script** — `NOTEBOOKLM_REFRESH_CMD` runs when auth has fully expired, then retries once.
|
||||
6. **Manual re-login** — `notebooklm login` (or `notebooklm login --master-token-refresh` to force a master-token re-mint; cold-dead cookies at process start need this).
|
||||
7. **External scheduler** — `notebooklm auth refresh` driven by cron / launchd / systemd / Task Scheduler / k8s CronJob, for idle profiles with no Python process running. Recommended cadence: 15–20 minutes.
|
||||
|
||||
> **Master-token troubleshooting:** `MasterTokenError: ... re-bootstrap` means the master token was revoked (password change / Google security action) — re-run `notebooklm login --master-token`. `... needs gpsoauth` means the `[headless]` extra isn't installed (`pip install "notebooklm-py[headless]"`). A minted jar "missing required cookies" indicates a MergeSession change — file an issue.
|
||||
>
|
||||
> **"This browser or app may not be secure" during `--master-token` sign-in:** Google blocks sign-in inside the automated browser the auto-capture launches. The client drops the obvious automation flags, but Google may still block — use one of the reliable paths instead:
|
||||
> - **Attach to your own Chrome (recommended):** quit Chrome, relaunch it with `--remote-debugging-port=9222`, then `notebooklm login --master-token --account you@gmail.com --cdp-url http://127.0.0.1:9222`. It opens an EmbeddedSetup tab in your real (non-automated) browser, so Google allows sign-in, and scrapes the `oauth_token`.
|
||||
> - **Capture the token manually:** in a normal browser sign in at `accounts.google.com/EmbeddedSetup`, copy the `oauth_token` cookie (DevTools → Application → Cookies → accounts.google.com), then `notebooklm login --master-token --account you@gmail.com --oauth-token <value>`. The `oauth_token` is single-use and short-lived — use it immediately.
|
||||
|
||||
Most users only need layer 1 — it's on by default and requires no configuration. For the full strategy (trade-offs between layers, including Python kwargs like `keepalive_min_interval` and environment variables like `NOTEBOOKLM_REFRESH_CMD_USE_SHELL`, and ready-to-paste launchd / systemd / cron / Task Scheduler / k8s CronJob recipes), see **[docs/auth-cookie-lifecycle.md#tldr](auth-cookie-lifecycle.md#tldr)** for a quick orientation, then [§4 The recovery ladder](auth-cookie-lifecycle.md#4--the-recovery-ladder) for the per-layer deep dive.
|
||||
|
||||
#### macOS: `--browser-cookies` prompts for your password
|
||||
|
||||
On macOS, Chrome (and Edge / Brave / Opera) encrypts its cookies file with a key stored in the **macOS Keychain** under the entry `Chrome Safe Storage`. By default that entry's ACL only allows `Google Chrome.app` itself to read the key without prompting; any other process — Python, Terminal, cron, an editor — gets a "wants to use the *Chrome Safe Storage* key" dialog. This is how macOS Keychain protects local data and applies to every cookie-extraction tool (`rookiepy`, `browser-cookie3`, `pycookiecheat`), not just `notebooklm-py`.
|
||||
|
||||
Workarounds, ordered by hassle:
|
||||
|
||||
1. **Click "Always Allow" in the prompt.** Adds the calling Python interpreter to the Keychain entry's ACL so subsequent runs of *that exact binary* should stop prompting. Caveat: rebuilding your venv (e.g. `uv venv` again) usually changes the interpreter path and you'll be re-prompted once for the new path.
|
||||
|
||||
2. **Use Touch ID instead of typing the password.** macOS Sonoma+ accepts Touch ID for Keychain dialogs — see *System Settings → Touch ID & Password*.
|
||||
|
||||
3. **Pre-unlock the login keychain in your shell** (best for cron jobs after one initial interactive run):
|
||||
```bash
|
||||
security unlock-keychain ~/Library/Keychains/login.keychain-db
|
||||
```
|
||||
Prompts once for your login password, then any process in the same login session can read entries you've already approved without re-prompting until the keychain auto-locks.
|
||||
|
||||
4. **Use Firefox as the cookie source.** Firefox stores cookies in a plain SQLite DB (no Keychain), so `notebooklm login --browser-cookies firefox` runs with **no prompt at all** — provided you're logged into Google in Firefox.
|
||||
```bash
|
||||
notebooklm login --browser-cookies firefox
|
||||
```
|
||||
This is the simplest answer for unattended macOS use.
|
||||
|
||||
**Firefox Multi-Account Containers note.** If your Google session
|
||||
lives inside a container, unscoped `--browser-cookies firefox` will
|
||||
merge cookies from every container into one jar (see issues
|
||||
[#366](https://github.com/teng-lin/notebooklm-py/issues/366) /
|
||||
[#367](https://github.com/teng-lin/notebooklm-py/issues/367)) and
|
||||
produce an inconsistent session. Use the explicit container syntax:
|
||||
```bash
|
||||
notebooklm login --browser-cookies 'firefox::Work' # named container
|
||||
notebooklm login --browser-cookies 'firefox::none' # no-container default
|
||||
```
|
||||
When a container is in use, the unscoped form also emits a yellow
|
||||
warning pointing at this syntax.
|
||||
|
||||
5. **Truly headless servers.** `--browser-cookies` is not the right tool — there's no live browser to extract from. Either re-extract on a workstation and ship `storage_state.json` to the server, or accept that human interaction is needed when cookies finally expire.
|
||||
|
||||
Quick diagnostic:
|
||||
```bash
|
||||
security find-generic-password -s 'Chrome Safe Storage' -a 'Chrome' -w >/dev/null && echo OK || echo "ACL or lock issue"
|
||||
```
|
||||
Prints `OK` without prompting → keychain is unlocked and your user has access; the prompt you saw is the per-binary ACL re-asking for a new caller (your Python). Click *Always Allow* once and that binary is permanently approved. If it prompts → run `security unlock-keychain` first.
|
||||
|
||||
#### Windows: `Missing required cookies: __Secure-1PSIDTS` after login, and `--browser-cookies` "Could not decrypt"
|
||||
|
||||
On Windows, both credential paths can leave you without `__Secure-1PSIDTS` (the rotating freshness partner of `__Secure-1PSID` that every real RPC needs — see [Automatic Token Refresh](#automatic-token-refresh) above), so `notebooklm login` reports success but `notebooklm list` then fails with `Missing required cookies: __Secure-1PSIDTS` (issue [#1753](https://github.com/teng-lin/notebooklm-py/issues/1753)). Two distinct causes are in play:
|
||||
|
||||
- **`notebooklm login --browser chrome` (Playwright flow).** The interactive browser completes Google sign-in, but Google may serve an automation-detected session *without* the token-binding cookie (and sometimes without the secondary-binding cookies `OSID` / `APISID` + `SAPISID` the automatic `RotateCookies` recovery needs to re-mint it). When that happens the saved `storage_state.json` is genuinely incomplete and re-running the same flow reproduces it.
|
||||
- **`notebooklm login --browser-cookies chrome` (or `edge`) → `Could not decrypt chrome cookies`.** Chrome 127+ (and current Edge) protect the cookie database with **App-Bound Encryption (ABE)**: the decryption key is bound to the browser process via a Windows service, so no external process can read it. This blocks every cookie-extraction library (`rookiepy`, `browser-cookie3`, `pycookiecheat`), not just `notebooklm-py`. There is no flag that bypasses ABE.
|
||||
|
||||
Note that `notebooklm doctor` may still say the auth check passed with an older client — the check historically only looked for `SID`. Current versions surface a **warn** row when `__Secure-1PSIDTS` is missing (`auth check --test` has always reported the real error). Trust `notebooklm auth check --test` / `notebooklm list` over a green `doctor` for "is this session actually usable".
|
||||
|
||||
Workarounds, most reliable first:
|
||||
|
||||
1. **Use Firefox as the cookie source.** Firefox stores cookies in a plain SQLite DB — **no App-Bound Encryption** — so extraction just works. Sign in to Google in Firefox, then:
|
||||
```bash
|
||||
notebooklm login --browser-cookies firefox
|
||||
```
|
||||
(If your Google session lives in a Multi-Account Containers tab, use the explicit `firefox::Container` / `firefox::none` syntax — see the [macOS section above](#macos---browser-cookies-prompts-for-your-password) for the container notes.) This is the simplest fix for the ABE case.
|
||||
|
||||
2. **Set up a master token (best for unattended / long-lived use).** `notebooklm login --master-token` (needs the `[headless]` extra: `pip install "notebooklm-py[headless]"`) stores a durable `master_token.json` beside your profile. When cookies are missing or fully expired, the client re-mints a complete, fresh cookie jar — including `__Secure-1PSIDTS` — from the master token in-process, so it does not depend on what the browser login happened to hand back. If Google blocks sign-in inside the automated capture window ("This browser or app may not be secure"), use the CDP-attach or manual `oauth_token` variants described in the [master-token troubleshooting note](#cookie-freshness-for-long-running--unattended-use) above. See also [installation.md#d-headless-server-or-ci](installation.md#d-headless-server-or-ci).
|
||||
|
||||
3. **Retry the Playwright login on a fresh profile.** Sometimes a stale persistent profile is the culprit rather than automation detection:
|
||||
```bash
|
||||
notebooklm login --fresh
|
||||
```
|
||||
If three attempts (normal, `--fresh` + password, `--fresh` + passkey) all reproduce the missing cookie, treat it as the automation-detection case and switch to Firefox or a master token above.
|
||||
|
||||
#### "Unauthorized" or redirect to login page
|
||||
|
||||
**Cause:** Session cookies expired (happens every few weeks).
|
||||
|
||||
**Note:** Automatic token refresh handles CSRF/session ID expiration. This error only occurs when the underlying cookies (set during `notebooklm login`) have fully expired.
|
||||
|
||||
**Solution:**
|
||||
```bash
|
||||
notebooklm login
|
||||
```
|
||||
|
||||
#### "Failed to extract CSRF token (SNlM0e)" / "CSRF token not found in HTML"
|
||||
|
||||
**Cause:** The CSRF token (`SNlM0e`) couldn't be extracted from the NotebookLM page response. The exact wording depends on which code path raised it:
|
||||
|
||||
- `Failed to extract CSRF token (SNlM0e). Page structure may have changed or authentication expired. Preview: '...'` — raised by `refresh_auth()` when the WIZ_global_data extraction fails ([`client.py`](../src/notebooklm/client.py)).
|
||||
- `CSRF token not found in HTML. Final URL: <url> This may indicate the page structure has changed.` — raised by the lower-level extractor when no auth redirect was detected ([`auth.py`](../src/notebooklm/auth.py)).
|
||||
- `Failed to extract 'SNlM0e' from NotebookLM HTML response. This usually means Google changed the page structure. Preview: '...'` — raised as `AuthExtractionError` directly (rare; usually wrapped by one of the messages above) ([`exceptions.py`](../src/notebooklm/exceptions.py)).
|
||||
|
||||
A related auth-redirect message — `Authentication expired. Run 'notebooklm login' to re-authenticate.` (or `Authentication expired or invalid. ...`) — surfaces the same root cause when the page redirected to Google's login flow.
|
||||
|
||||
**Note:** These errors should rarely surface, since the client automatically retries with a fresh CSRF token on auth failures (see *Automatic Token Refresh* above). When one does reach you, the automatic refresh also failed.
|
||||
|
||||
**Solution (if auto-refresh fails):**
|
||||
```python
|
||||
# In Python — manual refresh
|
||||
await client.refresh_auth()
|
||||
```
|
||||
Or re-run `notebooklm login` if session cookies are also expired. If the failure persists across re-login, the page structure has likely changed — file an issue and include the `Preview:` snippet from the error.
|
||||
|
||||
#### "NotebookLM redirected this request to its region / anti-abuse access gate"
|
||||
|
||||
**Cause:** The request to `notebooklm.google.com` was redirected to **`notebooklm.google/?location=unsupported`** — Google's region / anti-abuse risk-control gate (the marketing/landing page, which has no CSRF token). This is **not** a library bug, expired login, or page-structure change, and **re-running `notebooklm login` will not fix it** (the cookies are fine). It is driven by the *access environment*, not just the account's country, and fires even for accounts in supported regions when Google sees:
|
||||
|
||||
- a **VPN / proxy / datacenter / shared IP** (especially previously-abused ones),
|
||||
- an **IP ↔ timezone ↔ browser-language mismatch**, or
|
||||
- a **non-browser / automated access pattern** (a raw HTTP client without a real browser fingerprint).
|
||||
|
||||
**Confirm:** open `https://notebooklm.google.com` in a normal browser, signed in to the same account, on the same network. If it also redirects to `notebooklm.google/?location=unsupported`, the gate is environmental.
|
||||
|
||||
**Solution:** access from a **residential connection in a supported region**, keep your system **timezone/language consistent** with the IP's country, and avoid shared/datacenter VPN exit IPs. When the trigger is the **non-browser fingerprint** (a raw HTTP client) rather than the IP, the opt-in browser-TLS-impersonation transport can help: set `NOTEBOOKLM_TRANSPORT=curl_cffi` (requires the `curl_cffi` package) so requests carry a real browser's TLS fingerprint. (See issue [#1630](https://github.com/teng-lin/notebooklm-py/issues/1630).)
|
||||
|
||||
#### Browser opens but login fails
|
||||
|
||||
**Cause:** Google detecting automation and blocking login.
|
||||
|
||||
**Solution:**
|
||||
1. Delete the browser profile: `rm -rf ~/.notebooklm/profiles/<profile>/browser_profile/` (or `~/.notebooklm/profiles/default/browser_profile/` for the default profile)
|
||||
2. Run `notebooklm login` again
|
||||
3. Complete any CAPTCHA or security challenges Google presents
|
||||
4. Ensure you're using a real mouse/keyboard (not pasting credentials via script)
|
||||
|
||||
#### "Login not detected within 5 minutes" (especially on macOS)
|
||||
|
||||
**Cause:** The bundled Chromium that `notebooklm login` launches by default opened a fresh, signed-out browser, and its login-detection wait timed out — common on macOS where bundled Chromium can also be flaky (macOS 15+).
|
||||
|
||||
**Solution:** If you are already signed in to Google in **system Chrome**, retry with that browser so the existing session is reused instead of starting a fresh sign-in:
|
||||
|
||||
```bash
|
||||
notebooklm login --browser chrome --storage <path>
|
||||
```
|
||||
|
||||
`--browser chrome` drives your installed Google Chrome (with its signed-in profile), which usually detects the account immediately and sidesteps bundled-Chromium issues. `--browser msedge` is the equivalent for organizations that require Microsoft Edge for SSO.
|
||||
|
||||
### RPC Errors
|
||||
|
||||
#### "RPCError: No result found for RPC ID: XyZ123"
|
||||
|
||||
**Cause:** The RPC method ID may have changed (Google updates these periodically), or:
|
||||
- Rate limiting from Google
|
||||
- Account quota exceeded
|
||||
- API restrictions
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Enable debug mode to see what RPC IDs the server returns
|
||||
NOTEBOOKLM_DEBUG_RPC=1 notebooklm <your-command>
|
||||
```
|
||||
|
||||
This will show output like:
|
||||
```
|
||||
DEBUG: Looking for RPC ID: Ljjv0c
|
||||
DEBUG: Found RPC IDs in response: ['NewId123']
|
||||
```
|
||||
|
||||
If the IDs don't match, the method ID has changed. Report the new ID in a GitHub issue.
|
||||
|
||||
**Workaround:**
|
||||
- Wait 5-10 minutes and retry
|
||||
- Try with fewer sources selected
|
||||
- Reduce generation frequency
|
||||
|
||||
#### RPC method ID rotated by Google — self-patch with `NOTEBOOKLM_RPC_OVERRIDES`
|
||||
|
||||
Google rotates undocumented batchexecute method IDs without warning. When
|
||||
this happens, `notebooklm-py` raises `UnknownRPCMethodError` with the new ID
|
||||
the server now uses (see the previous section's diagnosis recipe). Rather
|
||||
than waiting for a release, you can patch the mapping for your process with
|
||||
the `NOTEBOOKLM_RPC_OVERRIDES` environment variable.
|
||||
|
||||
**Format:** JSON object mapping `RPCMethod` member names (the Python enum
|
||||
member name, not the obfuscated value) to the override RPC ID:
|
||||
|
||||
```bash
|
||||
export NOTEBOOKLM_RPC_OVERRIDES='{"LIST_NOTEBOOKS": "newId123", "CREATE_NOTEBOOK": "newId456"}'
|
||||
notebooklm list
|
||||
```
|
||||
|
||||
Or in Python:
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ["NOTEBOOKLM_RPC_OVERRIDES"] = '{"LIST_NOTEBOOKS": "newId123"}'
|
||||
|
||||
from notebooklm import NotebookLMClient
|
||||
# Subsequent client calls send the override IDs on the wire.
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
|
||||
- The override is applied at BOTH the URL `rpcids=` query parameter AND the
|
||||
request body `f.req` payload, so the wire format stays consistent.
|
||||
- The override is gated on the configured base host being a known Google
|
||||
NotebookLM endpoint (`notebooklm.google.com` or
|
||||
`notebooklm.cloud.google.com`). Overrides do NOT apply to non-Google
|
||||
hosts, so this env var cannot be weaponised to leak custom RPC IDs to a
|
||||
hostile endpoint.
|
||||
- Method names not listed in the override map continue to use the canonical
|
||||
IDs from `notebooklm.rpc.types.RPCMethod`.
|
||||
- Malformed input (invalid JSON, top-level array, etc.) is logged at
|
||||
`WARNING` and treated as no overrides.
|
||||
- The first time a distinct override set is applied in a process, the
|
||||
mapping is logged at `INFO` so you can confirm the config you intended is
|
||||
live.
|
||||
|
||||
**Discovering the new ID:** see the `NOTEBOOKLM_DEBUG_RPC=1` recipe above —
|
||||
the `Found RPC IDs in response: [...]` line tells you what the server is
|
||||
now returning. Cross-reference against the call site that failed.
|
||||
|
||||
Please also report the rotated IDs in a GitHub issue so the canonical
|
||||
mapping in `src/notebooklm/rpc/types.py` can be updated for everyone.
|
||||
|
||||
#### How to get the full response preview from an RPCError
|
||||
|
||||
`RPCError.raw_response` is truncated to **80 chars + `"..."`** by default so
|
||||
error messages stay readable in logs and CLI output. When you need the
|
||||
full body to diagnose schema drift or a malformed response, opt in:
|
||||
|
||||
```bash
|
||||
NOTEBOOKLM_DEBUG=1 notebooklm <your-command>
|
||||
```
|
||||
|
||||
Or in Python, set the env var before instantiating the client:
|
||||
|
||||
```python
|
||||
import os
|
||||
os.environ["NOTEBOOKLM_DEBUG"] = "1"
|
||||
|
||||
from notebooklm import NotebookLMClient
|
||||
# Subsequent RPCError instances will carry the full untruncated body.
|
||||
```
|
||||
|
||||
The value must be exactly `"1"` — `"0"`, `"true"`, etc. are treated as
|
||||
unset (still truncated).
|
||||
|
||||
#### "RPCError: [3]" (Invalid argument) / "UserDisplayableError"
|
||||
|
||||
**Cause:** Google's API rejected the request. Common cases:
|
||||
- Invalid parameters or a not-found resource ID
|
||||
- Account quota exceeded (for `create`, status `[3]` is also the notebook-limit signal)
|
||||
- Rate limiting
|
||||
|
||||
**Solution:**
|
||||
- Check that notebook/source IDs are valid
|
||||
- Add delays between operations (see Rate Limiting section)
|
||||
|
||||
**If it only affects _write_ operations** (`create`, `source add`, `generate`) while reads (`list`, `ask`) keep working — **and the web UI still works** — the likely cause is that Google changed the request payload (wire format) for those RPCs and `notebooklm-py` is still sending the old shape. Google rolls these out gradually, so it can hit some accounts before others.
|
||||
|
||||
> **First, rule out a mis-decoded success.** Re-run the failing action, then check `notebooklm list`: if the resource was actually **created** despite the error, it's a *response*-decoding issue (share the **Response** below). If it was **not** created, Google rejected our **request** (share the **Payload** below).
|
||||
|
||||
**Help us fix it — share the web UI's payload (no cookies needed).** Either option below leaks nothing: cookies, the `at=` CSRF token, and `Set-Cookie` live in request/response *headers*, never inside the `f.req` payload.
|
||||
|
||||
> **⚠️ Never paste the raw `.har` itself.** A HAR contains your cookies, the `at=` CSRF token, and the full NotebookLM page HTML — which embeds API keys, the CSRF token, and your account email. Only ever share the **scrubber's output** below (or the single `f.req` line from Option B). The scrubber processes only `/batchexecute` calls and redacts every value, so the page HTML never reaches its output.
|
||||
|
||||
**Option A — thorough, auto-scrubbed (recommended; captures every RPC + its response).**
|
||||
|
||||
This walkthrough takes ~2 minutes. You never copy a cookie, a token, or the raw HAR — a small bundled script does the redaction for you.
|
||||
|
||||
1. **Open DevTools on the Network tab.** In Chrome/Edge press <kbd>F12</kbd> (or <kbd>Cmd</kbd>+<kbd>Opt</kbd>+<kbd>I</kbd> on macOS); in Firefox press <kbd>F12</kbd>. Click the **Network** tab at the top of the panel.
|
||||
2. **Arm the capture.** Tick **Preserve log** (Chrome) / **Persist Logs** (Firefox) so a page reload doesn't wipe the capture, and confirm the round **● Record** button is red (it usually is by default).
|
||||
3. **Reproduce the failure.** In the NotebookLM tab, perform the exact action that fails — e.g. create a notebook, or add a source. You'll see `batchexecute?rpcids=…` rows appear in the Network list. You can stop as soon as the action errors.
|
||||
4. **Export the session to a file:**
|
||||
- **Chrome/Edge:** click the ⤓ **Export HAR…** download icon in the Network toolbar (or right-click any row → **Save all as HAR with content**).
|
||||
- **Firefox:** click the ⚙️ gear / **…** menu in the Network toolbar → **Save All As HAR**.
|
||||
|
||||
Save it as `capture.har`. (The "with content" variant matters — it's what includes the *response* bodies the scrubber reports.)
|
||||
5. **Scrub it.** From your `notebooklm-py` checkout, run the bundled script (stdlib-only — no install needed):
|
||||
|
||||
```console
|
||||
$ python scripts/scrub_rpc_har.py capture.har
|
||||
NotebookLM RPC capture — string values → <str:N>; cookies / headers / at= / Set-Cookie never read:
|
||||
|
||||
CCqFvf (CREATE_NOTEBOOK)
|
||||
request : ["<str:7>",null,null,[2],[1]]
|
||||
response: HTTP 200 | status_code=[3] | result=null
|
||||
|
||||
1 call(s). Safe to share — no cookies / CSRF / session tokens are present (they live in headers, which this tool never reads).
|
||||
```
|
||||
|
||||
Narrow to a single RPC with `--rpcid` if the capture is noisy:
|
||||
|
||||
```console
|
||||
$ python scripts/scrub_rpc_har.py capture.har --rpcid CCqFvf
|
||||
```
|
||||
|
||||
The script reads **only** each request's `f.req` field and the response body — never the headers/cookies arrays, never the non-`batchexecute` page HTML — and replaces every text value with its length (`<str:7>`). It refuses to print if any raw string ever slips through, so the output is safe by construction.
|
||||
6. **Paste that output** into the issue. Read it back first as a sanity check: every value should be `<str:N>`, never readable text.
|
||||
|
||||
- `status_code=[3]` with `result=null` → Google rejected our **request** (a payload/wire-format change). This is what we need to fix it.
|
||||
- A non-null `result` → the call actually worked and this is a **response**-decode issue; share it just the same.
|
||||
|
||||
**Option B — quick, one RPC by hand (no script).** Use this if you can't run the script.
|
||||
|
||||
1. In DevTools → **Network**, click the `batchexecute?rpcids=…` POST for the failing call (e.g. `rpcids=CCqFvf` for create, `izAoDd` for add-source).
|
||||
2. Open the **Payload** tab (Chrome) / **Request** tab (Firefox), copy **only** the `f.req` value — **not** the `at=` field beside it, and don't open the **Cookies**/**Headers** tabs.
|
||||
3. Replace any free text (title, URL) with `REDACTED` and paste it. We diff it against what the library sends — `["<title>", null, null, [2], [1]]` for create — and update the payload.
|
||||
|
||||
### Generation Failures
|
||||
|
||||
#### Audio/Video generation is refused immediately
|
||||
|
||||
**Cause:** NotebookLM refused the generation kickoff synchronously (often quota,
|
||||
feature availability, rate limiting, or an RPC shape drift). In v0.8.0 the
|
||||
Python API raises instead of returning `None`.
|
||||
|
||||
**What to do:**
|
||||
```bash
|
||||
# Let the CLI surface the typed error envelope / message
|
||||
notebooklm generate audio --wait --json
|
||||
|
||||
# If generation was accepted and you have a task id, poll manually
|
||||
notebooklm artifact poll <task_id> --json
|
||||
```
|
||||
|
||||
In Python, catch `RateLimitError`, `ArtifactFeatureUnavailableError`, or
|
||||
`RPCError` depending on the failure. If kickoff succeeds and later polling
|
||||
times out, use the timeout guidance below.
|
||||
|
||||
#### Audio/Video task times out as pending or in progress
|
||||
|
||||
**Cause:** NotebookLM accepted the generation task, but the upstream media queue
|
||||
did not reach a terminal state before your wait budget. For media artifacts, the
|
||||
SDK also keeps polling if NotebookLM marks the row completed before the media
|
||||
URL is populated.
|
||||
|
||||
**Solution:**
|
||||
- Increase the wait budget with `--timeout` or the Python
|
||||
`wait_for_completion(..., timeout=...)` argument.
|
||||
- For `generate <kind> --wait`, the built-in media defaults are 1200s for
|
||||
audio, 1800s for standard video, and 3600s for cinematic video; pass a
|
||||
larger `--timeout` if your account's media queue is slower.
|
||||
- `artifact wait` is intentionally generic and still defaults to 300s; when
|
||||
waiting manually on a media task ID, pass the matching media timeout.
|
||||
- Catch `ArtifactPendingTimeoutError` to retry queued tasks separately from
|
||||
`ArtifactInProgressTimeoutError`, which means the task started but did not
|
||||
finish before the timeout.
|
||||
- Log `exc.status_history` and `exc.status_transitions` for upstream queueing
|
||||
diagnostics instead of parsing the exception message.
|
||||
|
||||
#### Mind map or data table "generates" but doesn't appear
|
||||
|
||||
**Cause:** Generation may silently fail without error.
|
||||
|
||||
**Solution:**
|
||||
- Wait 60 seconds and check `artifact list`
|
||||
- Try regenerating with different/fewer sources
|
||||
|
||||
### File Upload Issues
|
||||
|
||||
#### HTML/XHTML files are rejected before upload
|
||||
|
||||
**Cause:** NotebookLM's file-upload endpoint rejects HTML-family uploads, even
|
||||
though the web UI may accept pasted rich text.
|
||||
|
||||
**Solution:** Convert saved web pages to text, Markdown, or PDF before adding
|
||||
them with your preferred extractor:
|
||||
|
||||
```bash
|
||||
notebooklm source add ./article.txt
|
||||
```
|
||||
|
||||
You can also pipe extracted text through stdin:
|
||||
|
||||
```bash
|
||||
python extract_article_text.py ./article.html | notebooklm source add - --type text --title "Article"
|
||||
```
|
||||
|
||||
#### Text/Markdown upload succeeds but processing/content is wrong
|
||||
|
||||
**Cause:** The upload was accepted, but NotebookLM processed unexpected content
|
||||
or reported a source-processing error. Current `add_file()` returns a `Source`;
|
||||
missing or untrusted source IDs raise `SourceAddError` instead of returning
|
||||
`None`.
|
||||
|
||||
**Workaround:** When you control the text, bypass file-type inference and use
|
||||
`add_text`:
|
||||
```bash
|
||||
# Instead of: notebooklm source add ./notes.txt
|
||||
# Do:
|
||||
notebooklm source add - --type text --title "My Notes" < ./notes.txt
|
||||
```
|
||||
|
||||
Or in Python:
|
||||
```python
|
||||
content = Path("notes.txt").read_text()
|
||||
await client.sources.add_text(nb_id, "My Notes", content)
|
||||
```
|
||||
|
||||
#### Large files time out
|
||||
|
||||
**Cause:** Files over ~20MB may exceed upload timeout.
|
||||
|
||||
**Solution:** Split large documents or use text extraction locally.
|
||||
|
||||
---
|
||||
|
||||
### Protected Website Content Issues
|
||||
|
||||
#### X.com / Twitter content incorrectly parsed as error page
|
||||
|
||||
**Symptoms:**
|
||||
- Source title shows "Fixing X.com Privacy Errors" or similar error message
|
||||
- Generated content discusses browser extensions instead of the actual article
|
||||
- Source appears to process successfully but contains wrong content
|
||||
|
||||
**Cause:** X.com (Twitter) has aggressive anti-scraping protections. When NotebookLM attempts to fetch the URL, it receives an error page or compatibility warning instead of the actual content.
|
||||
|
||||
**Solution - Use `bird` CLI to pre-fetch content:**
|
||||
|
||||
The `bird` CLI can fetch X.com content and output clean markdown:
|
||||
|
||||
```bash
|
||||
# Step 1: Install bird (macOS/Linux)
|
||||
brew install steipete/tap/bird
|
||||
|
||||
# Step 2: Fetch X.com content as markdown
|
||||
bird read "https://x.com/username/status/1234567890" > article.md
|
||||
|
||||
# Step 3: Add the local markdown file to NotebookLM
|
||||
notebooklm source add ./article.md
|
||||
```
|
||||
|
||||
**Alternative methods:**
|
||||
|
||||
**Using browser automation:**
|
||||
```bash
|
||||
# If you have playwright/browser-use available
|
||||
# Fetch content via browser and save as markdown
|
||||
```
|
||||
|
||||
**Manual extraction:**
|
||||
1. Open the X.com post in a browser
|
||||
2. Copy the text content
|
||||
3. Save to a `.md` file
|
||||
4. Add the file to NotebookLM
|
||||
|
||||
**Verification:**
|
||||
|
||||
Always verify the source was correctly parsed:
|
||||
```bash
|
||||
notebooklm source list
|
||||
# Check that the title matches the actual article, not an error message
|
||||
```
|
||||
|
||||
If the title contains error-related text, remove the source and use the pre-fetch method:
|
||||
```bash
|
||||
# Remove incorrectly parsed source
|
||||
notebooklm source delete <source_id>
|
||||
# Or, if you only have the exact title:
|
||||
notebooklm source delete-by-title "Exact Source Title"
|
||||
|
||||
# Then re-add using the bird CLI method above
|
||||
```
|
||||
|
||||
**Other affected sites:**
|
||||
- Some paywalled news sites
|
||||
- Sites requiring JavaScript execution for content
|
||||
- Sites with aggressive bot detection
|
||||
|
||||
---
|
||||
|
||||
## Known Limitations
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
Google enforces strict rate limits on the batchexecute endpoint.
|
||||
|
||||
**Symptoms:**
|
||||
- `RateLimitError` in Python, or CLI JSON with `code: "RATE_LIMITED"`
|
||||
- `RPCError` with ID `R7cb6c`
|
||||
- `UserDisplayableError` with code `[3]`
|
||||
|
||||
**Best Practices:**
|
||||
|
||||
**CLI:** Use `--retry` for automatic exponential backoff:
|
||||
```bash
|
||||
notebooklm generate audio --retry 3 # Retry up to 3 times on rate limit
|
||||
notebooklm generate video --retry 5 # Works with most generate commands
|
||||
```
|
||||
|
||||
*Note: `generate mind-map` is synchronous and does not accept the `--retry` option. All other `generate` subcommands support `--retry`.*
|
||||
|
||||
**Python:**
|
||||
```python
|
||||
import asyncio
|
||||
from notebooklm import RPCError
|
||||
from notebooklm.artifacts import with_rate_limit_retry
|
||||
|
||||
# Add delays between intensive operations
|
||||
for url in urls:
|
||||
await client.sources.add_url(nb_id, url)
|
||||
await asyncio.sleep(2) # 2 second delay
|
||||
|
||||
# Use the shared generation retry policy when starting artifacts
|
||||
status = await with_rate_limit_retry(
|
||||
lambda: client.artifacts.generate_audio(nb_id),
|
||||
max_retries=3,
|
||||
)
|
||||
|
||||
# For non-artifact RPC calls, retry by passing a fresh callable each attempt
|
||||
async def retry_rpc_call(make_call, max_retries=3):
|
||||
for attempt in range(max_retries + 1):
|
||||
try:
|
||||
return await make_call()
|
||||
except RPCError:
|
||||
if attempt >= max_retries:
|
||||
raise
|
||||
await asyncio.sleep(2**attempt)
|
||||
|
||||
notebook = await retry_rpc_call(lambda: client.notebooks.create("Research Notes"))
|
||||
```
|
||||
|
||||
### Starting a brand-new conversation (resolves the older issue #659 workaround)
|
||||
|
||||
`client.chat.ask(notebook_id, question)` with `conversation_id=None`
|
||||
attaches the question to the user's **current** conversation on the
|
||||
notebook — by design. The SDK still fetches the server-recorded
|
||||
conversation_id via `hPTbtc` after the ask and returns it on
|
||||
`AskResult.conversation_id`, so follow-ups using that id work
|
||||
correctly.
|
||||
|
||||
To force a brand-new server-side conversation, delete the existing
|
||||
one first — this mirrors the web UI's "Delete history" button:
|
||||
|
||||
```python
|
||||
last_conv_id = await client.chat.get_conversation_id(nb_id)
|
||||
if last_conv_id:
|
||||
await client.chat.delete_conversation(nb_id, last_conv_id)
|
||||
result = await client.chat.ask(nb_id, "Start fresh")
|
||||
```
|
||||
|
||||
Or via the CLI (prompts for confirmation; `-y` skips):
|
||||
|
||||
```bash
|
||||
notebooklm ask --new -y "Start fresh"
|
||||
```
|
||||
|
||||
**This is destructive: deleted turns are not recoverable.** The CLI
|
||||
shows the conversation's short id in the prompt and defaults to "No".
|
||||
`--json` implies `--yes` so scripted callers don't hang on stdin.
|
||||
|
||||
**History:** Before the SDK gained `delete_conversation` it had no way
|
||||
to honor the "fresh conversation" intent — both the SDK and the CLI's
|
||||
`--new` flag would silently extend the most-recent conversation, so
|
||||
users worked around it by creating a new notebook for each thread.
|
||||
The `J7Gthc` RPC was reverse-engineered from the web UI's "Delete
|
||||
history" button and removes the need for that workaround.
|
||||
|
||||
### Quota Restrictions
|
||||
|
||||
Some features have daily/hourly quotas:
|
||||
- **Audio Overviews:** Limited generations per day per account
|
||||
- **Video Overviews:** More restricted than audio
|
||||
- **Deep Research:** Consumes significant backend resources
|
||||
|
||||
### Download Requirements
|
||||
|
||||
Artifact downloads (audio, video, images) use `httpx` with cookies from your storage state. **Playwright is NOT required for downloads**—only for the initial `notebooklm login`.
|
||||
|
||||
If downloads fail with authentication errors:
|
||||
|
||||
**Solution:** Ensure your authentication is valid:
|
||||
```bash
|
||||
# Re-authenticate if cookies have expired
|
||||
notebooklm login
|
||||
|
||||
# Or copy a fresh storage_state.json from another machine
|
||||
```
|
||||
|
||||
**Custom auth paths:** When using `from_storage(path=...)` or `from_storage(profile="work")`,
|
||||
artifact downloads automatically use the same storage path for cookie authentication.
|
||||
If you are on an older version where downloads fail with "Storage file not found" pointing
|
||||
to the default location, upgrade or set `NOTEBOOKLM_HOME` as a workaround.
|
||||
|
||||
### URL Expiry
|
||||
|
||||
Download URLs for audio/video are temporary:
|
||||
- Expire within hours
|
||||
- Always fetch fresh URLs before downloading:
|
||||
|
||||
```python
|
||||
# Get fresh artifact list before download
|
||||
artifacts = await client.artifacts.list(nb_id)
|
||||
audio = next(a for a in artifacts if a.kind == "audio")
|
||||
# Use audio.url immediately
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Platform-Specific Issues
|
||||
|
||||
### Linux
|
||||
|
||||
**Playwright missing dependencies:**
|
||||
```bash
|
||||
playwright install-deps chromium
|
||||
```
|
||||
|
||||
**`playwright install chromium` fails with `TypeError: onExit is not a function`:**
|
||||
|
||||
This is an environment-specific Playwright install failure that has been observed with some newer Playwright builds on Linux. `notebooklm-py` only needs a working browser install for `notebooklm login`; the workaround is to install a known-good Playwright version in a clean virtual environment.
|
||||
|
||||
**Workaround** (intentionally uses `pip` rather than the canonical `uv sync --frozen` flow from [installation.md#e-contributor](installation.md#e-contributor) — this workaround needs to *override* the `playwright>=1.40.0` constraint to a specific older version, which `uv sync --frozen` would refuse):
|
||||
```bash
|
||||
python -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -U pip
|
||||
pip install "playwright==1.57.0"
|
||||
python -m playwright install chromium
|
||||
pip install -e ".[all]"
|
||||
```
|
||||
|
||||
**Why this order matters:**
|
||||
- `python -m playwright ...` ensures you use the Playwright module from the active virtual environment
|
||||
- installing the browser before `pip install -e ".[all]"` avoids picking up an older broken global `playwright` executable
|
||||
- if you already have another `playwright` on your system, verify with `which playwright` after activation
|
||||
- using `pip` here (not `uv sync --frozen`) is deliberate: this workaround needs to override the project's resolved `playwright` version with a specific older release, which the locked `uv` flow would block
|
||||
|
||||
If you need a non-editable install from Git instead of a local checkout, replace the last step with:
|
||||
```bash
|
||||
pip install "git+https://github.com/<your-user>/notebooklm-py@<branch>"
|
||||
```
|
||||
|
||||
**No display available (headless server):**
|
||||
- Browser login requires a display
|
||||
- Authenticate on a machine with GUI, then copy `storage_state.json`
|
||||
|
||||
### macOS
|
||||
|
||||
**Chromium not opening:**
|
||||
```bash
|
||||
# Re-install Playwright browsers
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
**Security warning about Chromium:**
|
||||
- Allow in System Preferences → Security & Privacy
|
||||
|
||||
### Windows
|
||||
|
||||
**CLI hangs indefinitely (issue #75):**
|
||||
|
||||
On certain Windows environments (particularly when running inside Sandboxie or similar sandboxing software), the CLI may hang indefinitely at startup. This is caused by the default `ProactorEventLoop` blocking at the IOCP (I/O Completion Ports) layer.
|
||||
|
||||
**Symptoms:**
|
||||
- CLI starts but never responds
|
||||
- Process appears frozen with no output
|
||||
- Happens consistently in sandboxed environments
|
||||
|
||||
**Solution:** The library automatically sets `WindowsSelectorEventLoopPolicy` at CLI startup to avoid this issue. If you're using the Python API directly and encounter hanging, add this before any async code:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import sys
|
||||
|
||||
if sys.platform == "win32":
|
||||
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
|
||||
```
|
||||
|
||||
**Unicode encoding errors on non-English Windows (issue #75, #80):**
|
||||
|
||||
Windows systems with non-English locales (Chinese cp950, Japanese cp932, etc.) may fail with `UnicodeEncodeError` when outputting Unicode characters like checkmarks (✓) or emojis.
|
||||
|
||||
**Symptoms:**
|
||||
- `UnicodeEncodeError: 'cp950' codec can't encode character`
|
||||
- Error occurs when printing status output with Rich tables
|
||||
|
||||
**Solution:** The library automatically sets `PYTHONUTF8=1` at CLI startup. For Python API usage, either:
|
||||
1. Set `PYTHONUTF8=1` environment variable before running
|
||||
2. Run Python with `-X utf8` flag: `python -X utf8 your_script.py`
|
||||
|
||||
**Path issues:**
|
||||
- Use forward slashes or raw strings: `r"C:\path\to\file"`
|
||||
- Ensure `~` expansion works: use `Path.home()` in Python
|
||||
|
||||
### WSL
|
||||
|
||||
**Browser opens in Windows, not WSL:**
|
||||
- This is expected behavior
|
||||
- Storage file is saved in WSL filesystem
|
||||
|
||||
---
|
||||
|
||||
## Debugging Tips
|
||||
|
||||
### Logging Configuration
|
||||
|
||||
`notebooklm-py` provides structured logging to help debug issues. The variables below are the logging-relevant subset; for the full environment-variable reference (storage, profile, network, decoder strictness, RPC overrides, etc.) and precedence rules, see [docs/configuration.md#environment-variables](configuration.md#environment-variables).
|
||||
|
||||
**Environment Variables (logging-specific):**
|
||||
|
||||
| Variable | Default | Effect |
|
||||
|----------|---------|--------|
|
||||
| `NOTEBOOKLM_LOG_LEVEL` | `WARNING` | Set to `DEBUG`, `INFO`, `WARNING`, or `ERROR` |
|
||||
| `NOTEBOOKLM_DEBUG_RPC` | (unset) | Legacy: Set to `1` to enable `DEBUG` level |
|
||||
| `NOTEBOOKLM_DEBUG` | (unset) | Set to `1` to preserve the full raw RPC response body on `RPCError.raw_response` (default: truncated to 80 chars + `"..."`) |
|
||||
|
||||
**When to use each level:**
|
||||
|
||||
```bash
|
||||
# WARNING (default): Only show warnings and errors
|
||||
notebooklm list
|
||||
|
||||
# INFO: Show major operations (good for scripts/automation)
|
||||
NOTEBOOKLM_LOG_LEVEL=INFO notebooklm source add https://example.com
|
||||
# Output:
|
||||
# 14:23:45 INFO [notebooklm._sources] Adding URL source: https://example.com
|
||||
|
||||
# DEBUG: Show all RPC calls with timing (for troubleshooting API issues)
|
||||
NOTEBOOKLM_LOG_LEVEL=DEBUG notebooklm list
|
||||
# Output:
|
||||
# 14:23:45 DEBUG [notebooklm._core] RPC LIST_NOTEBOOKS starting
|
||||
# 14:23:46 DEBUG [notebooklm._core] RPC LIST_NOTEBOOKS completed in 0.842s
|
||||
```
|
||||
|
||||
**Programmatic use:**
|
||||
|
||||
```python
|
||||
import logging
|
||||
import os
|
||||
|
||||
# Set before importing notebooklm
|
||||
os.environ["NOTEBOOKLM_LOG_LEVEL"] = "DEBUG"
|
||||
|
||||
from notebooklm import NotebookLMClient
|
||||
# Now all notebooklm operations will log at DEBUG level
|
||||
```
|
||||
|
||||
### Test Basic Operations
|
||||
|
||||
Start simple to isolate issues:
|
||||
|
||||
```bash
|
||||
# 1. Can you list notebooks?
|
||||
notebooklm list
|
||||
|
||||
# 2. Can you create a notebook?
|
||||
notebooklm create "Test"
|
||||
|
||||
# 3. Can you add a source?
|
||||
notebooklm source add "https://example.com"
|
||||
```
|
||||
|
||||
### Network Debugging
|
||||
|
||||
If you suspect network issues:
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
# Test basic connectivity
|
||||
async with httpx.AsyncClient() as client:
|
||||
r = await client.get("https://notebooklm.google.com")
|
||||
print(r.status_code) # Should be 200 or 302
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adapter-specific issues (MCP server, REST server)
|
||||
|
||||
The MCP and REST servers have their own setup and failure modes documented with
|
||||
the features:
|
||||
|
||||
- **MCP server / remote connector** (stdio & HTTP transports, self-hosted OAuth,
|
||||
Cloudflare / Tailscale tunnels): see [mcp-guide.md#troubleshooting](mcp-guide.md#troubleshooting)
|
||||
and [deploy/README.md](../deploy/README.md).
|
||||
- **REST server** (`notebooklm-server`): a non-loopback bind refuses to start
|
||||
without `NOTEBOOKLM_SERVER_TOKEN`; every `/v1` request needs the bearer (401
|
||||
otherwise). See [installation.md#rest-api-server](installation.md#rest-api-server).
|
||||
- **`curl_cffi` transport**: `NOTEBOOKLM_TRANSPORT=curl_cffi` requires the
|
||||
`curl_cffi` package; see the region / anti-abuse gate section above.
|
||||
|
||||
## Getting Help
|
||||
|
||||
1. Check this troubleshooting guide
|
||||
2. Search [existing issues](https://github.com/teng-lin/notebooklm-py/issues)
|
||||
3. Open a new issue with:
|
||||
- Command/code that failed
|
||||
- Full error message
|
||||
- Python version (`python --version`)
|
||||
- Library version (`notebooklm --version`)
|
||||
- Operating system
|
||||
@@ -0,0 +1,464 @@
|
||||
# Upgrading to v0.8.0
|
||||
|
||||
**Status:** Active
|
||||
**Last Updated:** 2026-07-04
|
||||
|
||||
`notebooklm-py` v0.8.0 **landed** a batch of **breaking** error-and-return
|
||||
contract changes under [ADR-0019](adr/0019-error-and-return-contract.md) (umbrella
|
||||
[#1346](https://github.com/teng-lin/notebooklm-py/issues/1346)). The guiding
|
||||
principle: *a return value encodes only success and genuine async-lifecycle
|
||||
state; resource absence, server refusal, and shape-drift **raise**.* `None`,
|
||||
`""`, empty sentinels, `"not_found"` strings, and `ValueError` are no longer used
|
||||
to signal that an error happened.
|
||||
|
||||
This guide is the single consolidated reference for moving your code across the
|
||||
0.7.0 → 0.8.0 boundary. Each migration below shows the **forward-compatible**
|
||||
form, which works on **both** 0.7.0 and 0.8.0 — so you can adopt it before
|
||||
upgrading and cross the boundary with no flag day.
|
||||
|
||||
For the canonical deprecation registry, see
|
||||
[docs/deprecations.md](deprecations.md). For the design rationale, see
|
||||
[ADR-0019](adr/0019-error-and-return-contract.md).
|
||||
|
||||
---
|
||||
|
||||
## Migrating from 0.7.x
|
||||
|
||||
> **`NOTEBOOKLM_FUTURE_ERRORS` is gone.** The v0.7.0 forward-compat preview flag
|
||||
> that let you opt into the v0.8.0 contract early was **removed in v0.8.0** now
|
||||
> that every break it staged is the default; setting it is a no-op. If you wired
|
||||
> it into CI, drop it.
|
||||
|
||||
If you are still on 0.7.x, drive your migration off the per-change
|
||||
`DeprecationWarning`s (the ✅ rows in the [summary table](#summary-table)). Python
|
||||
**hides `DeprecationWarning` by default** outside `__main__`, so enable them
|
||||
explicitly to surface the runways:
|
||||
|
||||
```bash
|
||||
python -W error::DeprecationWarning -m pytest # turn them into errors
|
||||
PYTHONWARNINGS=default::DeprecationWarning python app.py # just print them
|
||||
```
|
||||
|
||||
The ❌ rows in the summary table are **silent** clean breaks (return-value flips,
|
||||
refusal-raises, mutate-existing fail-loud) that emit no `DeprecationWarning` — for
|
||||
those, apply the forward-compatible migration below and verify against 0.8.0
|
||||
directly. `NOTEBOOKLM_QUIET_DEPRECATIONS=1` still silences the warnings while you
|
||||
migrate.
|
||||
|
||||
---
|
||||
|
||||
## Summary table
|
||||
|
||||
| Change | Warned in 0.7.0? | Migration (forward-compatible) | Broke in |
|
||||
|--------|------------------|--------------------------------|----------|
|
||||
| `sources` / `artifacts` / `notes` / `mind_maps` `.get()` returns `None` on a miss | ✅ on a miss | `get_or_none()`, or `try/except *NotFoundError` | v0.8.0 |
|
||||
| Dict-subscript (`result["key"]`) on typed research / mind-map / source-guide returns | ✅ on subscript | Attribute access (`result.status`, `guide.summary`) | v0.8.0 |
|
||||
| `research.wait_for_completion(interval=...)` | ✅ on use | `initial_interval=...` | v0.8.0 |
|
||||
| `generate mind-map` default `--kind` flips note-backed → interactive | ✅ stderr notice | Pass `--kind note-backed` (or `--kind interactive`) explicitly | v0.8.0 |
|
||||
| `NotebooksAPI.share()` removed | ✅ on use | `client.sharing.set_public(...)` | v0.8.0 |
|
||||
| `research.poll`/`wait_for_completion` `task_id=None` with ≥2 in-flight tasks | ✅ on ambiguity | Pass `task_id=` from `research.start` | v0.8.0 |
|
||||
| Synchronous generation refusal returns `GenerationStatus(status="failed")` | ❌ silent | `try/except RateLimitError` (or `with_rate_limit_retry`) | v0.8.0 |
|
||||
| `notes.update` / `rename(return_object=False)` silently no-op on a missing target | ❌ silent | `try/except *NotFoundError` | v0.8.0 |
|
||||
| `sources.refresh` / `chat.delete_conversation` return `bool` (always `True`) | ❌ silent | Stop relying on the return value | v0.8.0 |
|
||||
| `client.settings.get_account_tier()` and the `AccountTier` type removed | ❌ silent | `client.settings.get_account_limits()` (`AccountLimits.notebook_limit` / `source_limit`) — the old tier could not tell free from paid. **In v0.9.0 a correct `AccountLimits.tier` returns**, read from the authoritative limits block (not the promotions RPC) | v0.8.0 |
|
||||
| Derived-read / lister drift (`sources.check_freshness()`, note & artifact listers) returned an empty value on an unrecognized payload | ❌ silent | Handle `DecodingError` (only fires on server-side schema drift; legitimate empty/stale shapes are unchanged) | v0.8.0 |
|
||||
|
||||
Legend: ✅ emitted a `DeprecationWarning` (or a stderr notice) in 0.7.0; ❌ was a
|
||||
**silent** clean break with no v0.7.0 warning.
|
||||
|
||||
---
|
||||
|
||||
## 1. `get()` raises `*NotFoundError` instead of returning `None`
|
||||
|
||||
**What changes.** `sources.get()`, `artifacts.get()`, `notes.get()`, and
|
||||
`mind_maps.get()` raise the matching `*NotFoundError` on a miss instead of
|
||||
returning `None`, unifying them with `notebooks.get()` (which already raises).
|
||||
|
||||
**Warning in 0.7.0** (fires only on a miss):
|
||||
|
||||
> `DeprecationWarning: sources.get() returning None on a miss is deprecated and
|
||||
> will raise SourceNotFoundError in v0.8.0; use get_or_none() for None-on-miss.`
|
||||
|
||||
**Migration.** Two forward-compatible paths, both valid on 0.7.0 **and** 0.8.0:
|
||||
|
||||
```python
|
||||
# BEFORE — relies on the None-on-miss return (warns on a miss in 0.7.0; raises in 0.8.0)
|
||||
src = await client.sources.get(nb_id, source_id)
|
||||
if src is None:
|
||||
... # not found
|
||||
|
||||
# AFTER (a) — want None on a miss, warning-free: use get_or_none()
|
||||
src = await client.sources.get_or_none(nb_id, source_id)
|
||||
if src is None:
|
||||
... # not found
|
||||
|
||||
# AFTER (b) — want the raising contract: catch the typed exception
|
||||
from notebooklm import SourceNotFoundError
|
||||
try:
|
||||
src = await client.sources.get(nb_id, source_id)
|
||||
except SourceNotFoundError:
|
||||
... # not found
|
||||
```
|
||||
|
||||
`get_or_none()` exists on all four namespaces today (`sources`, `artifacts`,
|
||||
`notes`, `mind_maps`) and never warns. The `*NotFoundError` classes
|
||||
(`SourceNotFoundError`, `ArtifactNotFoundError`, `NoteNotFoundError`,
|
||||
`MindMapNotFoundError`) all exist today too — so the `try/except` form can be
|
||||
written now; it only starts *raising* in v0.8.0. All four are exported from
|
||||
both `notebooklm` and `notebooklm.exceptions`.
|
||||
|
||||
**Breaks in:** v0.8.0.
|
||||
Tracked by [#1247](https://github.com/teng-lin/notebooklm-py/issues/1247).
|
||||
|
||||
---
|
||||
|
||||
## 2. Typed returns drop dict-subscript access
|
||||
|
||||
**What changes.** `research.poll` / `research.start` /
|
||||
`research.wait_for_completion`, `artifacts.generate_mind_map`, and
|
||||
`sources.get_guide` return typed dataclasses (`ResearchTask`, `ResearchStart`,
|
||||
`MindMapResult`, `SourceGuide`). In 0.7.0 these still support legacy
|
||||
dict-subscript access (`result["key"]`) via a back-compat mixin; v0.8.0 drops
|
||||
the mixin and the returns become **attribute-only**.
|
||||
|
||||
**Warning in 0.7.0** (fires only on a `result["key"]` subscript):
|
||||
|
||||
> `DeprecationWarning: dict-style access on ResearchTask is deprecated; use
|
||||
> attribute access (e.g. result.status). Subscript access is removed in v0.8.0.`
|
||||
|
||||
Note that `result.get(...)`, `result.keys()`, `"x" in result`, and
|
||||
`iter(result)` stay **silent** in 0.7.0 — only `result["key"]` warns — but all
|
||||
of the mapping interface is removed in v0.8.0.
|
||||
|
||||
**Migration.** Switch subscripts to attribute access (works on both releases):
|
||||
|
||||
```python
|
||||
from notebooklm import ResearchStatus
|
||||
|
||||
# BEFORE — dict-subscript (warns in 0.7.0; removed in 0.8.0)
|
||||
result = await client.research.poll(nb_id)
|
||||
if result["status"] == "completed":
|
||||
for source in result["sources"]:
|
||||
print(source["title"], source["url"])
|
||||
|
||||
guide = await client.sources.get_guide(nb_id, src_id)
|
||||
print(guide["summary"], guide["keywords"])
|
||||
|
||||
# AFTER — typed attribute access (warning-free, valid on both releases)
|
||||
result = await client.research.poll(nb_id)
|
||||
if result.status == ResearchStatus.COMPLETED: # also == "completed"
|
||||
for source in result.sources: # tuple[ResearchSource, ...]
|
||||
print(source.title, source.url)
|
||||
|
||||
guide = await client.sources.get_guide(nb_id, src_id)
|
||||
print(guide.summary, guide.keywords) # guide.keywords is a tuple
|
||||
```
|
||||
|
||||
`ResearchStatus` is a `str` enum, so `result.status == "completed"` keeps
|
||||
working. The attributes map one-to-one onto the old keys: `ResearchTask` has
|
||||
`.task_id` / `.status` / `.query` / `.sources` / `.summary` / `.report` /
|
||||
`.tasks` (the sibling-tasks tuple from a top-level poll, formerly
|
||||
`result["tasks"]`); `ResearchSource` has `.title` / `.url`; `ResearchStart` has
|
||||
`.task_id`;
|
||||
`MindMapResult` has `.mind_map` / `.note_id`; `SourceGuide` has `.summary` /
|
||||
`.keywords`. The types are exported from both `notebooklm` and
|
||||
`notebooklm.types`.
|
||||
|
||||
**Breaks in:** v0.8.0.
|
||||
Tracked by [#1251](https://github.com/teng-lin/notebooklm-py/issues/1251)
|
||||
(follow-up to [#1209](https://github.com/teng-lin/notebooklm-py/issues/1209)).
|
||||
|
||||
---
|
||||
|
||||
## 3. `research.wait_for_completion(interval=...)` → `initial_interval=...`
|
||||
|
||||
**What changes.** The deprecated `interval=` keyword alias on
|
||||
`ResearchAPI.wait_for_completion` is removed; only `initial_interval=` is
|
||||
accepted. The rename aligns it with `SourcesAPI.wait_until_ready` and
|
||||
`ArtifactsAPI.wait_for_completion`, which already spell the cadence
|
||||
`initial_interval`.
|
||||
|
||||
**Warning in 0.7.0** (fires only when you pass a non-default `interval=`):
|
||||
|
||||
> `DeprecationWarning: the 'interval' keyword is deprecated; use
|
||||
> 'initial_interval' instead. 'interval' is removed in v0.8.0.`
|
||||
|
||||
Passing **both** `interval` and `initial_interval` raises `TypeError` today.
|
||||
|
||||
**Migration.** Rename the keyword (same cadence, valid on both releases):
|
||||
|
||||
```python
|
||||
# BEFORE — deprecated alias (warns in 0.7.0; removed in 0.8.0)
|
||||
await client.research.wait_for_completion(nb_id, task_id, interval=2.0)
|
||||
|
||||
# AFTER — canonical keyword, matches the source/artifact waiters
|
||||
await client.research.wait_for_completion(nb_id, task_id, initial_interval=2.0)
|
||||
```
|
||||
|
||||
**Breaks in:** v0.8.0.
|
||||
Tracked by [#1254](https://github.com/teng-lin/notebooklm-py/issues/1254)
|
||||
(follow-up to [#1208](https://github.com/teng-lin/notebooklm-py/issues/1208)).
|
||||
|
||||
---
|
||||
|
||||
## 4. `generate mind-map` default `--kind` flipped to interactive
|
||||
|
||||
**What changed (CLI).** `notebooklm generate mind-map` defaulted to the
|
||||
`note-backed` kind in 0.7.0; in v0.8.0 the default **flipped** to `interactive`
|
||||
(matching what NotebookLM's web app now creates). Note-backed stays available via
|
||||
an explicit `--kind note-backed` — it is **not** deprecated.
|
||||
|
||||
**Notice in 0.7.0** (printed to stderr when you don't pass `--kind`):
|
||||
|
||||
> `Note: 'generate mind-map' defaults to the note-backed kind today, but the
|
||||
> default switches to interactive in v0.8.0 (NotebookLM's web app already creates
|
||||
> interactive maps). Pass --kind note-backed or --kind interactive to pin your
|
||||
> choice; set NOTEBOOKLM_QUIET_DEPRECATIONS=1 to silence.`
|
||||
|
||||
**Migration.** Pin the kind explicitly so the default flip can't change your
|
||||
output shape (works identically on both releases):
|
||||
|
||||
```bash
|
||||
# BEFORE — relies on the default (note-backed today, interactive in v0.8.0)
|
||||
notebooklm generate mind-map -n <notebook-id>
|
||||
|
||||
# AFTER — pin your choice; behavior is stable across the version boundary
|
||||
notebooklm generate mind-map -n <notebook-id> --kind note-backed # JSON tree, synchronous
|
||||
notebooklm generate mind-map -n <notebook-id> --kind interactive # studio artifact, polled
|
||||
```
|
||||
|
||||
> **Python API note.** This is a **CLI-only** default change. The Python methods
|
||||
> are explicit and unaffected. Use the unified namespace when choosing a kind:
|
||||
> `client.mind_maps.generate(nb_id, kind=MindMapKind.INTERACTIVE)` returns the
|
||||
> interactive studio map, while `kind=MindMapKind.NOTE_BACKED` returns the
|
||||
> note-backed tree. The legacy `client.artifacts.generate_mind_map(...)` path is
|
||||
> note-backed.
|
||||
|
||||
**Breaks in:** v0.8.0.
|
||||
Tracked by [#1272](https://github.com/teng-lin/notebooklm-py/issues/1272).
|
||||
|
||||
---
|
||||
|
||||
## 5. `NotebooksAPI.share()` is removed
|
||||
|
||||
**What changes.** The deprecated `client.notebooks.share()` wrapper (a
|
||||
no-behavior-change shim, deprecated since v0.5.0) is removed.
|
||||
|
||||
**Warning in 0.7.0** (fires on every call):
|
||||
|
||||
> `DeprecationWarning: NotebooksAPI.share() is deprecated; use
|
||||
> client.sharing.set_public() (with add_user() / set_view_level() /
|
||||
> get_status()). Removed in v0.8.0.`
|
||||
|
||||
**Migration.** Call the `sharing` namespace directly (available on both
|
||||
releases):
|
||||
|
||||
```python
|
||||
# BEFORE — deprecated wrapper (warns in 0.7.0; removed in 0.8.0)
|
||||
await client.notebooks.share(nb_id, public=True)
|
||||
|
||||
# AFTER — the sharing namespace
|
||||
await client.sharing.set_public(nb_id, public=True)
|
||||
|
||||
# For the artifact deep-link URL (previously folded into share()):
|
||||
url = client.notebooks.get_share_url(nb_id, artifact_id)
|
||||
```
|
||||
|
||||
`set_public()` pairs with `add_user()`, `set_view_level()`, and `get_status()`
|
||||
for the full sharing surface; `get_share_url()` already exists today.
|
||||
|
||||
**Breaks in:** v0.8.0.
|
||||
Tracked by [#1363](https://github.com/teng-lin/notebooklm-py/issues/1363).
|
||||
|
||||
---
|
||||
|
||||
## 6. Ambiguous `research.poll` / `wait_for_completion` selection raises
|
||||
|
||||
**What changes.** `research.poll(nb_id)` and
|
||||
`research.wait_for_completion(nb_id)` keep `task_id` **optional**. With exactly
|
||||
one task in flight, `task_id=None` still returns it (no change). But with **two
|
||||
or more** tasks in flight and no `task_id`, v0.8.0 **raises** (a typed research
|
||||
error) instead of silently returning the *latest* task. This kills the
|
||||
silent-wrong-task hazard while keeping the single-task convenience.
|
||||
|
||||
**Warning in 0.7.0** (fires only when ≥2 tasks are in flight and no `task_id` is
|
||||
passed):
|
||||
|
||||
> `DeprecationWarning: research.poll() with multiple tasks in flight and no
|
||||
> task_id returns the latest task; pass task_id (from research.start) to select
|
||||
> one. This becomes an error in a future release.`
|
||||
|
||||
**Migration.** Capture the `task_id` from `research.start` and pass it through
|
||||
(works on both releases — single-task callers need no change):
|
||||
|
||||
```python
|
||||
# BEFORE — implicit "latest task" selection (warns with ≥2 in flight; raises in 0.8.0)
|
||||
await client.research.start(nb_id, query)
|
||||
result = await client.research.poll(nb_id)
|
||||
|
||||
# AFTER — pass the discriminator from start()
|
||||
started = await client.research.start(nb_id, query)
|
||||
result = await client.research.poll(nb_id, task_id=started.task_id)
|
||||
# ...and likewise:
|
||||
result = await client.research.wait_for_completion(nb_id, task_id=started.task_id)
|
||||
```
|
||||
|
||||
`research.start()` now returns `ResearchStart` or raises when the kickoff
|
||||
response is unusable; it no longer uses `None` to signal a couldn't-start
|
||||
payload.
|
||||
|
||||
**Breaks in:** v0.8.0.
|
||||
Tracked by [#1363](https://github.com/teng-lin/notebooklm-py/issues/1363).
|
||||
|
||||
---
|
||||
|
||||
## 7. Synchronous generation refusal raises (no more `status="failed"`)
|
||||
|
||||
**What changes.** When a generation kickoff
|
||||
(`generate_audio` / `generate_video` / `generate_mind_map` / `revise_slide` /
|
||||
…) is **refused synchronously** by the server (e.g. a rate limit /
|
||||
`USER_DISPLAYABLE_ERROR`), 0.7.0 *swallows* it into
|
||||
`GenerationStatus(status="failed")`. Per ADR-0019, a refusal is *couldn't-start*
|
||||
and v0.8.0 **raises** the `RateLimitError` / `RPCError` the decoder already
|
||||
produces. (A poll that observes a *started-then-failed* task still returns a
|
||||
terminal `failed` status — that is real async data, unchanged.)
|
||||
|
||||
**Warning in 0.7.0:** **None — this is a silent clean break.** There is no
|
||||
`DeprecationWarning`; apply the forward-compatible migration below and verify
|
||||
against 0.8.0 directly.
|
||||
|
||||
**Migration.** Catch `RateLimitError` (and/or `RPCError`) around the kickoff
|
||||
instead of inspecting a returned `status`. This pattern is valid on both
|
||||
releases — on 0.7.0 the refusal is still returned as `status="failed"`, so keep
|
||||
the status check too if you support both during the transition:
|
||||
|
||||
```python
|
||||
from notebooklm import RateLimitError
|
||||
|
||||
# BEFORE — inspects the returned status (only catches the swallowed refusal on 0.7.0)
|
||||
status = await client.artifacts.generate_audio(nb_id)
|
||||
if status.is_failed:
|
||||
... # may be a refusal OR a started-then-failed task — ambiguous
|
||||
|
||||
# AFTER — catch the refusal; a returned failed status is genuine async failure
|
||||
try:
|
||||
status = await client.artifacts.generate_audio(nb_id)
|
||||
except RateLimitError:
|
||||
... # synchronous refusal (couldn't start) — retry later / back off
|
||||
else:
|
||||
if status.is_failed:
|
||||
... # the task started and then reached a terminal failed state
|
||||
```
|
||||
|
||||
If you use the built-in retry helper, prefer it — it is rewritten for this
|
||||
contract and handles both the 0.7.0 returned-status and the 0.8.0 raised-error
|
||||
shapes:
|
||||
|
||||
```python
|
||||
from notebooklm.artifacts import with_rate_limit_retry
|
||||
|
||||
status = await with_rate_limit_retry(
|
||||
lambda: client.artifacts.generate_audio(nb_id),
|
||||
max_retries=3,
|
||||
)
|
||||
```
|
||||
|
||||
**Breaks in:** v0.8.0.
|
||||
Tracked by [#1342](https://github.com/teng-lin/notebooklm-py/issues/1342).
|
||||
|
||||
---
|
||||
|
||||
## 8. `notes.update` / `rename(return_object=False)` fail loud on a missing target
|
||||
|
||||
**What changes.** Mutate-existing operations against a **missing** target start
|
||||
raising `*NotFoundError` instead of silently no-op'ing:
|
||||
|
||||
- `notes.update(...)` on a non-existent note raises `NoteNotFoundError` (today it
|
||||
silently "succeeds").
|
||||
- `sources.rename(..., return_object=False)` and
|
||||
`artifacts.rename(..., return_object=False)` raise `SourceNotFoundError` /
|
||||
`ArtifactNotFoundError` on a missing target. The `return_object` flag now
|
||||
controls **hydration/return only**, not miss-detection — both modes raise on a
|
||||
missing target. (`return_object=True` already fails loud on 0.7.0;
|
||||
`mind_maps.rename` already fails loud in both modes.)
|
||||
|
||||
**Warning in 0.7.0:** **None — this is a silent clean break.** Apply the
|
||||
forward-compatible migration below and verify against 0.8.0 directly.
|
||||
|
||||
**Migration.** Wrap the mutation in `try/except *NotFoundError`. The classes
|
||||
exist today, so this is forward-compatible — on 0.7.0 the missing-target case is
|
||||
still a silent no-op (the `except` simply never fires); on 0.8.0 it catches the
|
||||
raise:
|
||||
|
||||
```python
|
||||
from notebooklm import NoteNotFoundError, SourceNotFoundError
|
||||
|
||||
# notes.update
|
||||
try:
|
||||
await client.notes.update(nb_id, note_id, "new content", "new title")
|
||||
except NoteNotFoundError:
|
||||
... # the note doesn't exist (silently no-op on 0.7.0; raises on 0.8.0)
|
||||
|
||||
# rename(return_object=False)
|
||||
try:
|
||||
await client.sources.rename(nb_id, source_id, "new title", return_object=False)
|
||||
except SourceNotFoundError:
|
||||
... # the source doesn't exist
|
||||
```
|
||||
|
||||
**Breaks in:** v0.8.0.
|
||||
Tracked by [#1362](https://github.com/teng-lin/notebooklm-py/issues/1362).
|
||||
|
||||
---
|
||||
|
||||
## 9. `sources.refresh` / `chat.delete_conversation` return `None`
|
||||
|
||||
**What changes.** `SourcesAPI.refresh` and `ChatAPI.delete_conversation` are
|
||||
annotated `-> bool` but only ever return `True` (any failure raises first). To
|
||||
remove the misleading signal — mirroring the `delete() -> None` cleanup already
|
||||
shipped on the 0.7.0 breaking line — they become `-> None` in v0.8.0.
|
||||
|
||||
> **No clean forward-compatible single-line form.** On 0.7.0 these return a
|
||||
> truthy `True`; on 0.8.0 they return `None` (falsy). Any code shaped like
|
||||
> `if await client.sources.refresh(...):` flips behavior across the boundary.
|
||||
> The fix is to **stop branching on the return value** — success is signalled by
|
||||
> *not raising*, not by the return — which is correct on both releases.
|
||||
|
||||
**Warning in 0.7.0:** **None — this is a silent clean break** (a return-type
|
||||
change with no `DeprecationWarning`). Grep for truthiness checks like
|
||||
`if await client.sources.refresh(...):` / `if await client.chat.delete_conversation(...):`
|
||||
to find affected code, and apply the migration below.
|
||||
|
||||
**Migration.** Drop the truthiness check; rely on the call not raising:
|
||||
|
||||
```python
|
||||
# BEFORE — branches on the always-True return (flips on 0.8.0, where it returns None)
|
||||
if await client.sources.refresh(nb_id, source_id):
|
||||
print("refreshed")
|
||||
|
||||
# AFTER — success == no exception; works on both releases
|
||||
await client.sources.refresh(nb_id, source_id)
|
||||
print("refreshed")
|
||||
|
||||
# Same shape for chat:
|
||||
await client.chat.delete_conversation(nb_id, conversation_id)
|
||||
```
|
||||
|
||||
> `ChatAPI.clear_cache()` is **not** changing — its `bool` return is meaningful
|
||||
> (it reports whether anything was cleared) and stays `-> bool`.
|
||||
|
||||
**Breaks in:** v0.8.0.
|
||||
Tracked by [#1290](https://github.com/teng-lin/notebooklm-py/issues/1290).
|
||||
|
||||
---
|
||||
|
||||
## See also
|
||||
|
||||
- [docs/deprecations.md](deprecations.md) — the single source of truth for
|
||||
deprecated APIs with removal-version pins.
|
||||
- [docs/adr/0019-error-and-return-contract.md](adr/0019-error-and-return-contract.md)
|
||||
— the design rationale for the whole convergence.
|
||||
- [docs/stability.md](stability.md) — the semver promise and 0.x pre-1.0
|
||||
deprecation policy.
|
||||
- [docs/configuration.md](configuration.md) — `NOTEBOOKLM_QUIET_DEPRECATIONS`
|
||||
and other environment variables.
|
||||
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Bulk import sources example.
|
||||
|
||||
This script demonstrates:
|
||||
1. Create a notebook
|
||||
2. Add multiple sources of different types
|
||||
3. Handle errors gracefully
|
||||
4. Report import status
|
||||
|
||||
Prerequisites:
|
||||
pip install "notebooklm-py[browser]" && playwright install chromium
|
||||
notebooklm login
|
||||
# Full install guide: https://github.com/teng-lin/notebooklm-py/blob/main/docs/installation.md
|
||||
|
||||
Usage:
|
||||
python bulk-import.py
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
from notebooklm import NotebookLMClient
|
||||
|
||||
# Example sources to import
|
||||
SOURCES = {
|
||||
"urls": [
|
||||
"https://en.wikipedia.org/wiki/Machine_learning",
|
||||
"https://en.wikipedia.org/wiki/Deep_learning",
|
||||
],
|
||||
"youtube": [
|
||||
"https://www.youtube.com/watch?v=aircAruvnKk", # 3Blue1Brown neural networks
|
||||
],
|
||||
"text": [
|
||||
{
|
||||
"title": "Project Notes",
|
||||
"content": """
|
||||
Key points for our ML research project:
|
||||
- Focus on transformer architectures
|
||||
- Compare with traditional RNN approaches
|
||||
- Benchmark on standard datasets
|
||||
""",
|
||||
},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
async def main():
|
||||
print("=== Bulk Import Example ===\n")
|
||||
|
||||
async with NotebookLMClient.from_storage() as client:
|
||||
# 1. Create a notebook
|
||||
print("Creating notebook...")
|
||||
nb = await client.notebooks.create("Bulk Import Demo")
|
||||
print(f" Created: {nb.id}\n")
|
||||
|
||||
results = {"success": [], "failed": []}
|
||||
|
||||
# 2. Import URLs
|
||||
print("Importing URLs...")
|
||||
for url in SOURCES["urls"]:
|
||||
try:
|
||||
source = await client.sources.add_url(nb.id, url)
|
||||
results["success"].append(f"URL: {source.title}")
|
||||
print(f" + {source.title}")
|
||||
except Exception as e:
|
||||
results["failed"].append(f"URL: {url} - {e}")
|
||||
print(f" - Failed: {url}")
|
||||
|
||||
# 3. Import YouTube videos (add_url auto-detects YouTube)
|
||||
print("\nImporting YouTube videos...")
|
||||
for url in SOURCES["youtube"]:
|
||||
try:
|
||||
source = await client.sources.add_url(nb.id, url)
|
||||
results["success"].append(f"YouTube: {source.title}")
|
||||
print(f" + {source.title}")
|
||||
except Exception as e:
|
||||
results["failed"].append(f"YouTube: {url} - {e}")
|
||||
print(f" - Failed: {url}")
|
||||
|
||||
# 4. Import text content
|
||||
print("\nImporting text content...")
|
||||
for item in SOURCES["text"]:
|
||||
try:
|
||||
source = await client.sources.add_text(nb.id, item["title"], item["content"])
|
||||
results["success"].append(f"Text: {source.title}")
|
||||
print(f" + {source.title}")
|
||||
except Exception as e:
|
||||
results["failed"].append(f"Text: {item['title']} - {e}")
|
||||
print(f" - Failed: {item['title']}")
|
||||
|
||||
# 5. Report results
|
||||
print("\n" + "=" * 40)
|
||||
print("Import complete!")
|
||||
print(f" Successful: {len(results['success'])}")
|
||||
print(f" Failed: {len(results['failed'])}")
|
||||
|
||||
if results["failed"]:
|
||||
print("\nFailed imports:")
|
||||
for item in results["failed"]:
|
||||
print(f" - {item}")
|
||||
|
||||
print(f"\n Notebook ID: {nb.id}")
|
||||
print(" (Notebook kept for review - delete manually when done)")
|
||||
|
||||
print("\n=== Done! ===")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,185 @@
|
||||
"""Example: Chat with a notebook and manage conversations.
|
||||
|
||||
This example demonstrates:
|
||||
1. Asking questions about notebook content
|
||||
2. Follow-up questions in a conversation
|
||||
3. Retrieving conversation history
|
||||
4. Configuring chat behavior (response length, custom personas)
|
||||
|
||||
Prerequisites:
|
||||
- Authentication configured via `notebooklm auth` CLI command
|
||||
- Valid Google account with NotebookLM access
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
from notebooklm import ChatGoal, ChatMode, ChatResponseLength, NotebookLMClient
|
||||
|
||||
|
||||
async def main():
|
||||
"""Demonstrate chat and conversation features."""
|
||||
|
||||
async with NotebookLMClient.from_storage() as client:
|
||||
# Create a notebook with some content
|
||||
print("Setting up notebook with sources...")
|
||||
notebook = await client.notebooks.create("Python Learning")
|
||||
|
||||
# Add a source for context
|
||||
source = await client.sources.add_url(
|
||||
notebook.id,
|
||||
"https://en.wikipedia.org/wiki/Python_(programming_language)",
|
||||
)
|
||||
print(f"Added source: {source.title}")
|
||||
|
||||
# Give NotebookLM a moment to process the source
|
||||
print("Waiting for source processing...")
|
||||
await asyncio.sleep(3)
|
||||
|
||||
# =====================================================================
|
||||
# Basic Question/Answer
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Basic Q&A ---")
|
||||
|
||||
# Ask a question about the notebook's content
|
||||
result = await client.chat.ask(
|
||||
notebook.id,
|
||||
"What are the main features of Python?",
|
||||
)
|
||||
|
||||
print("Question: What are the main features of Python?")
|
||||
print(f"Answer: {result.answer[:500]}...")
|
||||
print(f"Conversation ID: {result.conversation_id}")
|
||||
print(f"Turn number: {result.turn_number}")
|
||||
|
||||
# =====================================================================
|
||||
# Follow-up Questions (Conversation Threading)
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Follow-up Questions ---")
|
||||
|
||||
# Use the same conversation_id for follow-up questions
|
||||
# This maintains context from previous exchanges
|
||||
followup = await client.chat.ask(
|
||||
notebook.id,
|
||||
"How does it compare to other programming languages?",
|
||||
conversation_id=result.conversation_id, # Continue the conversation
|
||||
)
|
||||
|
||||
print("Follow-up: How does it compare to other programming languages?")
|
||||
print(f"Answer: {followup.answer[:500]}...")
|
||||
print(f"Is follow-up: {followup.is_follow_up}")
|
||||
print(f"Turn number: {followup.turn_number}")
|
||||
|
||||
# Another follow-up
|
||||
followup2 = await client.chat.ask(
|
||||
notebook.id,
|
||||
"What about for data science specifically?",
|
||||
conversation_id=result.conversation_id,
|
||||
)
|
||||
|
||||
print("\nFollow-up 2: What about for data science specifically?")
|
||||
print(f"Answer: {followup2.answer[:400]}...")
|
||||
|
||||
# =====================================================================
|
||||
# Conversation History
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Conversation History ---")
|
||||
|
||||
# Get locally cached conversation turns
|
||||
turns = client.chat.get_cached_turns(result.conversation_id)
|
||||
print(f"Cached turns in this conversation: {len(turns)}")
|
||||
for turn in turns:
|
||||
print(f" Turn {turn.turn_number}:")
|
||||
print(f" Q: {turn.query[:50]}...")
|
||||
print(f" A: {turn.answer[:50]}...")
|
||||
|
||||
# Get conversation history from the API (all conversations)
|
||||
try:
|
||||
history = await client.chat.get_history(notebook.id, limit=10)
|
||||
print(f"\nAPI conversation history: {type(history)}")
|
||||
except Exception as e:
|
||||
print(f"Note: History retrieval returned: {e}")
|
||||
|
||||
# =====================================================================
|
||||
# Configuring Chat Behavior
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Chat Configuration ---")
|
||||
|
||||
# Method 1: Use predefined chat modes
|
||||
# Available modes: DEFAULT, LEARNING_GUIDE, CONCISE, DETAILED
|
||||
print("Setting chat mode to LEARNING_GUIDE...")
|
||||
await client.chat.set_mode(notebook.id, ChatMode.LEARNING_GUIDE)
|
||||
|
||||
# Ask a question with the new mode
|
||||
learning_result = await client.chat.ask(
|
||||
notebook.id,
|
||||
"Explain decorators in Python",
|
||||
)
|
||||
print(f"Learning mode answer: {learning_result.answer[:400]}...")
|
||||
|
||||
# Method 2: Fine-grained configuration
|
||||
# ChatGoal: DEFAULT, CUSTOM, LEARNING_GUIDE
|
||||
# ChatResponseLength: SHORTER, DEFAULT, LONGER
|
||||
print("\nSetting custom chat configuration...")
|
||||
await client.chat.configure(
|
||||
notebook.id,
|
||||
goal=ChatGoal.DEFAULT,
|
||||
response_length=ChatResponseLength.SHORTER,
|
||||
)
|
||||
|
||||
concise_result = await client.chat.ask(
|
||||
notebook.id,
|
||||
"What is Python used for?",
|
||||
)
|
||||
print(f"Concise answer: {concise_result.answer[:300]}...")
|
||||
|
||||
# Method 3: Custom persona with specific instructions
|
||||
print("\nSetting custom persona...")
|
||||
await client.chat.configure(
|
||||
notebook.id,
|
||||
goal=ChatGoal.CUSTOM,
|
||||
response_length=ChatResponseLength.DEFAULT,
|
||||
custom_prompt="You are an experienced Python developer. "
|
||||
"Explain concepts with practical code examples. "
|
||||
"Focus on best practices and real-world usage.",
|
||||
)
|
||||
|
||||
custom_result = await client.chat.ask(
|
||||
notebook.id,
|
||||
"How should I handle errors in Python?",
|
||||
)
|
||||
print(f"Custom persona answer: {custom_result.answer[:500]}...")
|
||||
|
||||
# =====================================================================
|
||||
# Source-Specific Questions
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Source-Specific Questions ---")
|
||||
|
||||
# Get source IDs to target specific sources
|
||||
sources = await client.sources.list(notebook.id)
|
||||
if sources:
|
||||
source_ids = [sources[0].id]
|
||||
|
||||
# Ask about specific sources only
|
||||
targeted_result = await client.chat.ask(
|
||||
notebook.id,
|
||||
"Summarize the key points from this source",
|
||||
source_ids=source_ids, # Only use these sources for context
|
||||
)
|
||||
print(f"Targeted answer: {targeted_result.answer[:400]}...")
|
||||
|
||||
# =====================================================================
|
||||
# Cleanup
|
||||
# =====================================================================
|
||||
|
||||
# Clear conversation cache (optional)
|
||||
client.chat.clear_cache(result.conversation_id)
|
||||
print("\nConversation cache cleared")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,260 @@
|
||||
"""Example: Create and manage notes and mind maps.
|
||||
|
||||
This example demonstrates:
|
||||
1. Creating and updating notes
|
||||
2. Listing and searching notes
|
||||
3. Generating mind maps from sources
|
||||
4. Working with study materials (reports, quizzes)
|
||||
|
||||
Notes are user-created content, distinct from AI-generated artifacts.
|
||||
They persist in your notebook and can be exported.
|
||||
|
||||
Prerequisites:
|
||||
- Authentication configured via `notebooklm auth` CLI command
|
||||
- Valid Google account with NotebookLM access
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
|
||||
from notebooklm import NotebookLMClient, ReportFormat
|
||||
|
||||
|
||||
async def main():
|
||||
"""Demonstrate notes and mind map functionality."""
|
||||
|
||||
async with NotebookLMClient.from_storage() as client:
|
||||
# Create a notebook for our examples
|
||||
print("Creating notebook...")
|
||||
notebook = await client.notebooks.create("Study Notes Demo")
|
||||
print(f"Created notebook: {notebook.id}")
|
||||
|
||||
# Add a source for AI features to work with
|
||||
print("\nAdding source...")
|
||||
source = await client.sources.add_url(
|
||||
notebook.id,
|
||||
"https://en.wikipedia.org/wiki/Data_structure",
|
||||
)
|
||||
print(f"Added: {source.title}")
|
||||
|
||||
# Wait for source processing
|
||||
await asyncio.sleep(3)
|
||||
|
||||
# =====================================================================
|
||||
# Creating Notes
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Creating Notes ---")
|
||||
|
||||
# Create a new note with title and content
|
||||
note1 = await client.notes.create(
|
||||
notebook.id,
|
||||
title="Key Concepts",
|
||||
content="# Data Structures Overview\n\n"
|
||||
"- Arrays: Sequential memory storage\n"
|
||||
"- Linked Lists: Node-based storage\n"
|
||||
"- Trees: Hierarchical organization\n"
|
||||
"- Hash Tables: Key-value mapping",
|
||||
)
|
||||
print(f"Created note: {note1.title} (ID: {note1.id})")
|
||||
|
||||
# Create another note
|
||||
note2 = await client.notes.create(
|
||||
notebook.id,
|
||||
title="Study Questions",
|
||||
content="1. What is time complexity?\n"
|
||||
"2. When to use arrays vs linked lists?\n"
|
||||
"3. How do hash collisions work?",
|
||||
)
|
||||
print(f"Created note: {note2.title} (ID: {note2.id})")
|
||||
|
||||
# =====================================================================
|
||||
# Updating Notes
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Updating Notes ---")
|
||||
|
||||
# Update an existing note
|
||||
await client.notes.update(
|
||||
notebook.id,
|
||||
note1.id,
|
||||
content="# Data Structures Overview (Updated)\n\n"
|
||||
"## Linear Structures\n"
|
||||
"- Arrays: O(1) access, O(n) insertion\n"
|
||||
"- Linked Lists: O(n) access, O(1) insertion\n\n"
|
||||
"## Non-Linear Structures\n"
|
||||
"- Trees: Hierarchical, O(log n) search\n"
|
||||
"- Graphs: Network relationships",
|
||||
title="Key Concepts (Revised)",
|
||||
)
|
||||
print(f"Updated note: {note1.id}")
|
||||
|
||||
# =====================================================================
|
||||
# Listing Notes
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Listing Notes ---")
|
||||
|
||||
notes = await client.notes.list(notebook.id)
|
||||
print(f"Found {len(notes)} notes:")
|
||||
for note in notes:
|
||||
preview = note.content[:50].replace("\n", " ") if note.content else ""
|
||||
print(f" - {note.title}: {preview}...")
|
||||
|
||||
# =====================================================================
|
||||
# Getting a Specific Note
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Getting Specific Note ---")
|
||||
|
||||
# NOTE: get() returns the note, or None if it's missing. Returning None
|
||||
# on a miss is DEPRECATED — in v0.8.0 it will raise NoteNotFoundError
|
||||
# instead (issue #1247). To future-proof, prefer wrapping the call in
|
||||
# ``try/except NoteNotFoundError`` rather than the ``if retrieved:``
|
||||
# None-check shown here. See docs/deprecations.md.
|
||||
retrieved = await client.notes.get(notebook.id, note1.id)
|
||||
if retrieved:
|
||||
print(f"Title: {retrieved.title}")
|
||||
print(f"Content:\n{retrieved.content[:200]}...")
|
||||
|
||||
# =====================================================================
|
||||
# Mind Maps
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Generating Mind Map ---")
|
||||
|
||||
# Generate an interactive mind map from sources
|
||||
# This creates a visual representation of concepts
|
||||
try:
|
||||
mind_map_result = await client.artifacts.generate_mind_map(notebook.id)
|
||||
|
||||
if mind_map_result.get("mind_map"):
|
||||
print("Mind map generated successfully!")
|
||||
|
||||
# The mind map is returned as JSON data
|
||||
mind_map_data = mind_map_result["mind_map"]
|
||||
|
||||
# If it's a string, parse it
|
||||
if isinstance(mind_map_data, str):
|
||||
try:
|
||||
mind_map_data = json.loads(mind_map_data)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# Display mind map structure
|
||||
if isinstance(mind_map_data, dict):
|
||||
print(f"Root topic: {mind_map_data.get('label', 'N/A')}")
|
||||
children = mind_map_data.get("children", [])
|
||||
print(f"Main branches: {len(children)}")
|
||||
for child in children[:5]:
|
||||
if isinstance(child, dict):
|
||||
print(f" - {child.get('label', 'N/A')}")
|
||||
|
||||
if mind_map_result.get("note_id"):
|
||||
print(f"Mind map stored as note: {mind_map_result['note_id']}")
|
||||
else:
|
||||
print("Mind map generation returned empty result")
|
||||
except Exception as e:
|
||||
print(f"Mind map generation error: {e}")
|
||||
|
||||
# =====================================================================
|
||||
# Listing Mind Maps
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Listing Mind Maps ---")
|
||||
|
||||
mind_maps = await client.notes.list_mind_maps(notebook.id)
|
||||
print(f"Found {len(mind_maps)} mind maps")
|
||||
for mm in mind_maps:
|
||||
mm_id = mm[0] if isinstance(mm, list) and mm else "Unknown"
|
||||
print(f" - Mind map ID: {mm_id}")
|
||||
|
||||
# =====================================================================
|
||||
# Study Materials (Reports)
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Generating Study Materials ---")
|
||||
|
||||
# Generate a study guide
|
||||
print("Generating study guide...")
|
||||
study_gen = await client.artifacts.generate_study_guide(notebook.id)
|
||||
print(f"Study guide generation started: {study_gen.task_id}")
|
||||
|
||||
# Generate a briefing document
|
||||
print("Generating briefing doc...")
|
||||
briefing_gen = await client.artifacts.generate_report(
|
||||
notebook.id,
|
||||
report_format=ReportFormat.BRIEFING_DOC,
|
||||
)
|
||||
print(f"Briefing doc generation started: {briefing_gen.task_id}")
|
||||
|
||||
# Wait briefly and check status
|
||||
await asyncio.sleep(5)
|
||||
|
||||
# List all reports
|
||||
reports = await client.artifacts.list_reports(notebook.id)
|
||||
print(f"\nReports in notebook: {len(reports)}")
|
||||
for report in reports:
|
||||
status = "Ready" if report.is_completed else "Processing"
|
||||
print(f" - {report.title} ({status})")
|
||||
|
||||
# =====================================================================
|
||||
# Quizzes and Flashcards
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Generating Quiz ---")
|
||||
|
||||
from notebooklm import QuizDifficulty, QuizQuantity
|
||||
|
||||
quiz_gen = await client.artifacts.generate_quiz(
|
||||
notebook.id,
|
||||
quantity=QuizQuantity.STANDARD,
|
||||
difficulty=QuizDifficulty.MEDIUM,
|
||||
)
|
||||
print(f"Quiz generation started: {quiz_gen.task_id}")
|
||||
|
||||
# Generate flashcards
|
||||
print("Generating flashcards...")
|
||||
flashcard_gen = await client.artifacts.generate_flashcards(
|
||||
notebook.id,
|
||||
quantity=QuizQuantity.FEWER,
|
||||
difficulty=QuizDifficulty.EASY,
|
||||
)
|
||||
print(f"Flashcard generation started: {flashcard_gen.task_id}")
|
||||
|
||||
# =====================================================================
|
||||
# Deleting Notes
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Deleting Note ---")
|
||||
|
||||
# Delete a note
|
||||
success = await client.notes.delete(notebook.id, note2.id)
|
||||
print(f"Deleted note {note2.id}: {success}")
|
||||
|
||||
# Verify deletion
|
||||
remaining = await client.notes.list(notebook.id)
|
||||
print(f"Remaining notes: {len(remaining)}")
|
||||
|
||||
# =====================================================================
|
||||
# Summary
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Final Summary ---")
|
||||
|
||||
# List all artifacts
|
||||
all_artifacts = await client.artifacts.list(notebook.id)
|
||||
print(f"Total artifacts: {len(all_artifacts)}")
|
||||
|
||||
# Categorize by type using the user-facing kind property
|
||||
type_counts: dict[str, int] = {}
|
||||
for art in all_artifacts:
|
||||
type_name = art.kind.value # e.g., "audio", "video", "report"
|
||||
type_counts[type_name] = type_counts.get(type_name, 0) + 1
|
||||
|
||||
for type_name, count in type_counts.items():
|
||||
print(f" {type_name}: {count}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,75 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Quickstart example for notebooklm-py.
|
||||
|
||||
This script demonstrates a complete workflow:
|
||||
1. Create a notebook
|
||||
2. Add sources
|
||||
3. Chat with content
|
||||
4. Generate a podcast
|
||||
5. Download the result
|
||||
|
||||
Prerequisites:
|
||||
pip install "notebooklm-py[browser]" && playwright install chromium
|
||||
notebooklm login # Authenticate first
|
||||
# Full install guide: https://github.com/teng-lin/notebooklm-py/blob/main/docs/installation.md
|
||||
|
||||
Usage:
|
||||
python quickstart.py
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
from notebooklm import NotebookLMClient
|
||||
|
||||
|
||||
async def main():
|
||||
print("=== NotebookLM Quickstart ===\n")
|
||||
|
||||
async with NotebookLMClient.from_storage() as client:
|
||||
# 1. Create a notebook
|
||||
print("Creating notebook...")
|
||||
nb = await client.notebooks.create("Quickstart Demo")
|
||||
print(f" Created: {nb.id} - {nb.title}\n")
|
||||
|
||||
# 2. Add a source
|
||||
print("Adding source...")
|
||||
url = "https://en.wikipedia.org/wiki/Artificial_intelligence"
|
||||
source = await client.sources.add_url(nb.id, url)
|
||||
print(f" Added: {source.title}\n")
|
||||
|
||||
# 3. Chat with the content
|
||||
print("Asking a question...")
|
||||
result = await client.chat.ask(nb.id, "What are the main topics covered?")
|
||||
print(f" Answer: {result.answer[:200]}...\n")
|
||||
|
||||
# 4. Generate an audio overview
|
||||
print("Generating podcast (this may take a few minutes)...")
|
||||
status = await client.artifacts.generate_audio(
|
||||
nb.id, instructions="Focus on the history and key milestones"
|
||||
)
|
||||
print(f" Started generation, task_id: {status.task_id}")
|
||||
|
||||
# Wait for completion
|
||||
final = await client.artifacts.wait_for_completion(
|
||||
nb.id, status.task_id, timeout=300, initial_interval=10
|
||||
)
|
||||
|
||||
if final.is_complete:
|
||||
print(f" Complete! URL: {final.url}\n")
|
||||
|
||||
# 5. Download (requires browser support)
|
||||
# output_path = await client.artifacts.download_audio(nb.id, "./podcast.mp3")
|
||||
# print(f" Downloaded to: {output_path}")
|
||||
else:
|
||||
print(f" Generation status: {final.status}\n")
|
||||
|
||||
# Cleanup: Delete the demo notebook
|
||||
print("Cleaning up...")
|
||||
await client.notebooks.delete(nb.id)
|
||||
print(" Deleted demo notebook\n")
|
||||
|
||||
print("=== Done! ===")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Default ``NOTEBOOKLM_REFRESH_CMD`` script: re-extract cookies from a live browser.
|
||||
|
||||
Wires the ``NOTEBOOKLM_REFRESH_CMD`` auto-refresh hook to the existing
|
||||
``notebooklm login --browser-cookies`` flow so unattended automation can
|
||||
recover from auth expiry without a human at the terminal.
|
||||
|
||||
When auth fails (e.g. session fully expired, force-logout, password changed,
|
||||
or eventually a DBSC challenge that pure-httpx can't satisfy), the library
|
||||
runs this script, reloads ``storage_state.json``, and retries the original
|
||||
call once. The keepalive poke (``_poke_session``) handles routine SIDTS
|
||||
rotation while you're active; this script handles the harder case of "the
|
||||
session is gone, fetch a fresh one from a real browser."
|
||||
|
||||
Setup:
|
||||
pip install 'notebooklm-py[cookies]' # rookiepy for cookie extraction
|
||||
export NOTEBOOKLM_REFRESH_CMD="python /absolute/path/refresh_browser_cookies.py"
|
||||
|
||||
# Optional — pick a non-Chrome browser (chrome, edge, firefox, brave, ...)
|
||||
export NOTEBOOKLM_REFRESH_BROWSER=edge
|
||||
|
||||
The library injects ``NOTEBOOKLM_REFRESH_PROFILE`` and
|
||||
``NOTEBOOKLM_REFRESH_STORAGE_PATH`` into this script's environment so it
|
||||
targets the right profile / file.
|
||||
|
||||
Caveat:
|
||||
The source browser must already be logged into the same Google account.
|
||||
Cookie extraction reads whichever account is currently active in that
|
||||
browser — switch accounts in the browser before running if you need a
|
||||
different one for a given profile.
|
||||
"""
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
def main() -> int:
|
||||
missing = [
|
||||
var
|
||||
for var in ("NOTEBOOKLM_REFRESH_PROFILE", "NOTEBOOKLM_REFRESH_STORAGE_PATH")
|
||||
if not os.environ.get(var)
|
||||
]
|
||||
if missing:
|
||||
print(
|
||||
"refresh_browser_cookies.py: missing required environment variable(s): "
|
||||
f"{', '.join(missing)}. This script is intended to be invoked by the "
|
||||
"library via NOTEBOOKLM_REFRESH_CMD, not run directly. See the docstring "
|
||||
"for setup instructions.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
return 2
|
||||
|
||||
profile = os.environ["NOTEBOOKLM_REFRESH_PROFILE"]
|
||||
storage = os.environ["NOTEBOOKLM_REFRESH_STORAGE_PATH"]
|
||||
browser = os.environ.get("NOTEBOOKLM_REFRESH_BROWSER", "chrome")
|
||||
|
||||
# Use `python -m notebooklm.notebooklm_cli` instead of the `notebooklm` console
|
||||
# script so the script works inside venvs where the console script isn't on PATH
|
||||
# (e.g. cron jobs that don't source the venv's activate).
|
||||
# `--profile` is a top-level Click option and must come BEFORE the `login`
|
||||
# subcommand — Click rejects it after `login`.
|
||||
return subprocess.call(
|
||||
[
|
||||
sys.executable,
|
||||
"-m",
|
||||
"notebooklm.notebooklm_cli",
|
||||
"--profile",
|
||||
profile,
|
||||
"login",
|
||||
"--browser-cookies",
|
||||
browser,
|
||||
"--storage",
|
||||
storage,
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,97 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Research to podcast workflow example.
|
||||
|
||||
This script demonstrates:
|
||||
1. Create a notebook
|
||||
2. Start deep research on a topic
|
||||
3. Import discovered sources
|
||||
4. Generate a podcast from the research
|
||||
|
||||
Prerequisites:
|
||||
pip install "notebooklm-py[browser]" && playwright install chromium
|
||||
notebooklm login
|
||||
# Full install guide: https://github.com/teng-lin/notebooklm-py/blob/main/docs/installation.md
|
||||
|
||||
Usage:
|
||||
python research-to-podcast.py "Your research topic"
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
|
||||
from notebooklm import NotebookLMClient
|
||||
|
||||
|
||||
async def main(topic: str):
|
||||
print(f"=== Research to Podcast: {topic} ===\n")
|
||||
|
||||
async with NotebookLMClient.from_storage() as client:
|
||||
# 1. Create a notebook
|
||||
print("Creating notebook...")
|
||||
nb = await client.notebooks.create(f"Research: {topic}")
|
||||
print(f" Created: {nb.id}\n")
|
||||
|
||||
# 2. Start deep research
|
||||
print("Starting deep research (this may take a while)...")
|
||||
research = await client.research.start(nb.id, topic, source="web", mode="deep")
|
||||
task_id = research.get("task_id") if research else None
|
||||
print(f" Task ID: {task_id}\n")
|
||||
|
||||
# 3. Wait for completion
|
||||
print("Waiting for research to complete...")
|
||||
try:
|
||||
status = await client.research.wait_for_completion(
|
||||
nb.id,
|
||||
task_id=task_id,
|
||||
timeout=300,
|
||||
interval=10,
|
||||
)
|
||||
except TimeoutError:
|
||||
print(" Research timed out\n")
|
||||
return
|
||||
|
||||
if status.get("status") != "completed":
|
||||
print(f" Research ended with status: {status.get('status', 'unknown')}\n")
|
||||
return
|
||||
|
||||
task_id = status.get("task_id") or task_id
|
||||
sources = status.get("sources", [])
|
||||
print(f" Found {len(sources)} sources!\n")
|
||||
|
||||
# 4. Import discovered sources
|
||||
if sources and task_id:
|
||||
print("Importing sources...")
|
||||
await client.research.import_sources(nb.id, task_id, sources[:10]) # Limit to 10
|
||||
print(f" Imported {min(len(sources), 10)} sources\n")
|
||||
elif sources:
|
||||
print(" Skipping import: research completed without a task ID\n")
|
||||
|
||||
# 5. Generate podcast
|
||||
print("Generating podcast...")
|
||||
gen_status = await client.artifacts.generate_audio(
|
||||
nb.id, instructions=f"Create an engaging overview of {topic}"
|
||||
)
|
||||
|
||||
print("Waiting for audio generation...")
|
||||
final = await client.artifacts.wait_for_completion(nb.id, gen_status.task_id, timeout=600)
|
||||
|
||||
if final.is_complete:
|
||||
print(f"\n Success! Audio URL: {final.url}")
|
||||
print("\n Use 'notebooklm download audio' to save the file")
|
||||
else:
|
||||
print(f"\n Generation ended with status: {final.status}")
|
||||
|
||||
print(f"\n Notebook ID: {nb.id}")
|
||||
print(" (Notebook kept for review - delete manually when done)")
|
||||
|
||||
print("\n=== Done! ===")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: python research-to-podcast.py 'Your research topic'")
|
||||
print("Example: python research-to-podcast.py 'renewable energy trends 2024'")
|
||||
sys.exit(1)
|
||||
|
||||
topic = " ".join(sys.argv[1:])
|
||||
asyncio.run(main(topic))
|
||||
@@ -0,0 +1,163 @@
|
||||
"""Example: Generate a Video Overview from notebook sources.
|
||||
|
||||
This example demonstrates:
|
||||
1. Setting up a notebook with sources
|
||||
2. Generating a video overview with style options
|
||||
3. Checking artifact status
|
||||
4. Downloading the completed video
|
||||
|
||||
Video Overviews are animated explainer videos that summarize your
|
||||
notebook content with AI-generated narration and visuals.
|
||||
|
||||
Prerequisites:
|
||||
- Authentication configured via `notebooklm auth` CLI command
|
||||
- Valid Google account with NotebookLM access
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
|
||||
from notebooklm import NotebookLMClient, VideoFormat, VideoStyle
|
||||
|
||||
|
||||
async def main():
|
||||
"""Generate a video overview from notebook sources."""
|
||||
|
||||
async with NotebookLMClient.from_storage() as client:
|
||||
# Step 1: Create a notebook with content
|
||||
print("Creating notebook...")
|
||||
notebook = await client.notebooks.create("Video Demo Notebook")
|
||||
print(f"Created notebook: {notebook.id}")
|
||||
|
||||
# Add sources for video content
|
||||
print("\nAdding sources...")
|
||||
urls = [
|
||||
"https://en.wikipedia.org/wiki/Quantum_computing",
|
||||
]
|
||||
|
||||
for url in urls:
|
||||
source = await client.sources.add_url(notebook.id, url)
|
||||
print(f" Added: {source.title or url}")
|
||||
|
||||
# Wait for source processing
|
||||
print("\nWaiting for source processing...")
|
||||
await asyncio.sleep(5)
|
||||
|
||||
# Step 2: Generate the video overview
|
||||
# Video generation options:
|
||||
#
|
||||
# video_format:
|
||||
# - EXPLAINER: Full explanatory video (default)
|
||||
# - BRIEF: Shorter summary video
|
||||
# - CINEMATIC: AI-generated documentary footage (Veo 3)
|
||||
# - SHORT: Vertical short-form video (fixed style; no video_style)
|
||||
#
|
||||
# video_style:
|
||||
# - AUTO_SELECT: Let AI choose the best style (default)
|
||||
# - CUSTOM: Caller-provided style
|
||||
# - CLASSIC: Traditional presentation style
|
||||
# - WHITEBOARD: Whiteboard animation style
|
||||
# - KAWAII: Cute, kawaii-inspired style
|
||||
# - ANIME: Anime visual style
|
||||
# - WATERCOLOR: Watercolor painting style
|
||||
# - RETRO_PRINT: Retro print style
|
||||
# - HERITAGE: Heritage/vintage style
|
||||
# - PAPER_CRAFT: Paper-craft style
|
||||
|
||||
print("\nStarting video generation...")
|
||||
print("Video generation typically takes 3-8 minutes")
|
||||
|
||||
generation = await client.artifacts.generate_video(
|
||||
notebook.id,
|
||||
video_format=VideoFormat.EXPLAINER,
|
||||
video_style=VideoStyle.AUTO_SELECT,
|
||||
language="en",
|
||||
instructions="Create an engaging overview suitable for general audiences",
|
||||
)
|
||||
|
||||
print(f"Generation started: {generation.task_id}")
|
||||
print(f"Initial status: {generation.status}")
|
||||
|
||||
# Step 3: Wait for completion with status updates
|
||||
print("\nWaiting for video generation...")
|
||||
|
||||
try:
|
||||
final_status = await client.artifacts.wait_for_completion(
|
||||
notebook.id,
|
||||
generation.task_id,
|
||||
initial_interval=10.0, # Check every 10 seconds initially
|
||||
max_interval=30.0, # Max 30 seconds between checks
|
||||
timeout=900.0, # 15 minute timeout for videos
|
||||
)
|
||||
|
||||
if final_status.is_complete:
|
||||
print("\nVideo generation complete!")
|
||||
|
||||
# Step 4: Download the video
|
||||
output_path = "quantum_video.mp4"
|
||||
print(f"Downloading video to {output_path}...")
|
||||
|
||||
await client.artifacts.download_video(
|
||||
notebook.id,
|
||||
output_path,
|
||||
artifact_id=generation.task_id,
|
||||
)
|
||||
print(f"Video downloaded: {output_path}")
|
||||
|
||||
elif final_status.is_failed:
|
||||
print(f"\nGeneration failed: {final_status.error}")
|
||||
|
||||
except TimeoutError:
|
||||
print("\nVideo generation timed out")
|
||||
print("Check NotebookLM web UI for completion")
|
||||
|
||||
# =====================================================================
|
||||
# Alternative: Check existing artifacts
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Listing All Video Artifacts ---")
|
||||
|
||||
# List all video artifacts in the notebook
|
||||
videos = await client.artifacts.list_video(notebook.id)
|
||||
|
||||
for video in videos:
|
||||
status = "Ready" if video.is_completed else "Processing"
|
||||
print(f"\n Title: {video.title}")
|
||||
print(f" ID: {video.id}")
|
||||
print(f" Status: {status}")
|
||||
if video.created_at:
|
||||
print(f" Created: {video.created_at}")
|
||||
|
||||
# =====================================================================
|
||||
# Manual status polling example
|
||||
# =====================================================================
|
||||
|
||||
print("\n--- Manual Status Polling ---")
|
||||
|
||||
if generation.task_id:
|
||||
# You can manually poll status without wait_for_completion
|
||||
status = await client.artifacts.poll_status(
|
||||
notebook.id,
|
||||
generation.task_id,
|
||||
)
|
||||
print(f"Current status: {status.status}")
|
||||
print(f"Is complete: {status.is_complete}")
|
||||
print(f"Is in progress: {status.is_in_progress}")
|
||||
|
||||
# =====================================================================
|
||||
# Download existing video
|
||||
# =====================================================================
|
||||
|
||||
# If you have an existing completed video, download it directly
|
||||
if videos and videos[0].is_completed:
|
||||
print("\n--- Downloading Existing Video ---")
|
||||
existing_path = "existing_video.mp4"
|
||||
await client.artifacts.download_video(
|
||||
notebook.id,
|
||||
existing_path,
|
||||
artifact_id=videos[0].id,
|
||||
)
|
||||
print(f"Downloaded: {existing_path}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user