teng-lin--notebooklm-py/docs/refactor-history.md

Tier 12 / Tier 13 Refactor — Historical Record

Status: Shipped in v0.5.0. Current runtime shape: docs/architecture.md. Ratifying decision: ADR-0013 — composable session capabilities, which supersedes ADR-0010 — Session/Kernel split. Last updated: 2026-06-11

⚠️ Superseded in part by ADR-0014. Later refactors removed the Session facade class entirely and then promoted the flat _session_* / _runtime_* helpers into the current package homes: _runtime/contracts.py, _runtime/auth.py, _runtime/lifecycle.py, _runtime/config.py, _runtime/helpers.py, _runtime/transport.py, _runtime/init.py, and _middleware/*. The module names in the migration tables below are the v0.5.0 as-shipped names, not the current tree — see docs/architecture.md for the current module map.

This document is the historical record of the Tier 12 / Tier 13 refactor arc. It exists for two audiences:

• Maintainers who want to understand the design intent — why the feature-facing Session boundary was rebuilt around composable capability Protocols rather than a single broad Protocol — and what tradeoffs that locked in.
• First-party callers and test suites that imported underscore-prefixed (notebooklm._...) internals during Tier 11 or earlier, and need to migrate to the new module paths.

If your code only uses the documented public API (docs/python-api.md), nothing changed for you — the v0.4.1 public surface was a binding constraint on the whole arc.

Background

Tier 12 ("middleware chain") and Tier 13 ("Session/Kernel split") restructured the private core layer of notebooklm-py:

• Tier 12 extracted the authed HTTP path into a composable middleware chain (AuthedHttpClient Protocol, Middleware Protocol, RequestContext). Tracing, metrics, drain bookkeeping, error injection, 429 / 5xx retry, 401 auth-refresh, and the global RPC concurrency semaphore moved into discrete middleware modules and off of the monolithic transport class.
• Tier 13 split the ClientCore god-object into Session (the orchestrator) and Kernel (the pure transport core — owns the httpx.AsyncClient and the cookie jar). It also renamed every _core_* module out of the legacy _core namespace into final homes that later consolidated under _runtime/, _middleware/, _chat/, _request_types, _transport_errors, _streaming_post, _rpc_executor, and so on.
• The capability refactor that followed (ADR-0013) replaced the broad Session Protocol that Tier 13 originally shipped with a composable set of narrow capability Protocols (RpcCaller, LoopGuard, OperationScopeProvider, AsyncWorkRuntime) plus feature-local runtime Protocols (ChatRuntime, ArtifactsRuntime, UploadRuntime). Feature APIs now depend on the narrowest slice of capability they actually use, not on the orchestrator class. Later ADR-0014/#1327 cleanup tightened this further: the current code passes each concrete collaborator directly, keeps AuthMetadata and OperationScopeProvider local to their only consumers, and leaves only Kernel, RpcCaller, and LoopGuard in _runtime/contracts.py.

The public NotebookLMClient API — client.notebooks.*, client.sources.*, client.chat.*, client.artifacts.*, and friends — was preserved across all three movements.

What changed by audience

Downstream callers using the public API

Nothing. Every method, property, and attribute reachable through NotebookLMClient at v0.4.1 still works with the same signature and return type:

from notebooklm import NotebookLMClient, AuthTokens
from notebooklm.rpc import RPCMethod

async with NotebookLMClient.from_storage() as client:
    notebooks = await client.notebooks.list()
    await client.sources.add_url(notebook_id, url)
    result = await client.chat.ask(notebook_id, question)
    status = await client.artifacts.generate_audio(notebook_id)

One additive surface from this arc was client.chat.save_answer_as_note(...), which is now the canonical citation-rich saved-from-chat workflow. The transitional client.notes.create_from_chat(...) forwarder was removed in v0.7.0.

See docs/stability.md for the public stability contract.

First-party callers and test suites

Underscore-prefixed names are not part of the documented public surface and may move again in future tiers. The tables below are provided as a courtesy for first-party callers and test suites that imported them during Tier 11 or earlier.

Quick guidance:

• The notebooklm._core compatibility shim and the NotebookLMClient._core attribute alias were both deleted in Phase 4 (#889). Downstream code and tests must import from the current owning modules documented in docs/architecture.md.
• Prefer the public surface — notebooklm.NotebookLMClient, notebooklm.AuthTokens, notebooklm.rpc.RPCMethod, and the types / exceptions re-exported from the top-level package.
• Feature APIs (NotebooksAPI, SourcesAPI, ArtifactsAPI, ChatAPI, ResearchAPI, NotesAPI, MindMapsAPI, SharingAPI, SettingsAPI, LabelsAPI) depend on the narrow capability Protocols in notebooklm._runtime.contracts (Kernel, RpcCaller, LoopGuard) or on single-consumer Protocols defined in their owning modules — not on the deleted concrete Session class and not on a broad Session Protocol.

Renamed modules

The Tier 12 _core_* module paths no longer resolve. import notebooklm._core_auth (and every other row below) now raises ImportError. Update your imports to the new path on the right:

Tier 12 path	Tier 13 path
notebooklm._core_auth	notebooklm._runtime.auth
notebooklm._core_cache	notebooklm._conversation_cache
notebooklm._core_constants	notebooklm._runtime.config
notebooklm._core_cookie_persistence	notebooklm._cookie_persistence
notebooklm._core_drain	notebooklm._transport_drain
notebooklm._core_error_injection	notebooklm._error_injection
notebooklm._core_helpers	notebooklm._runtime.helpers
notebooklm._core_lifecycle	notebooklm._runtime.lifecycle
notebooklm._core_metrics	notebooklm._client_metrics
notebooklm._core_polling	notebooklm._polling_registry
notebooklm._core_reqid	notebooklm._reqid_counter
notebooklm._core_rpc	notebooklm._rpc_executor
notebooklm._core_transport	notebooklm._request_types / notebooklm._transport_errors / notebooklm._streaming_post

notebooklm._core itself has been deleted.

Moved and renamed symbols

The Session/Kernel split introduced new home modules for the orchestrator and its transport collaborator. Existing helper names did not change, only their home module did.

Tier 12 symbol	Tier 13 home	Notes
notebooklm._core.ClientCore (class)	Direct collaborators + notebooklm._runtime.contracts	ClientCore was retired. Feature APIs accept direct collaborators or the narrow shared Protocols in notebooklm._runtime.contracts (Kernel, RpcCaller, LoopGuard); the broad Session Protocol and _session_contracts module were retired — see ADR-0013.
notebooklm._core.MAX_RETRY_AFTER_SECONDS	notebooklm._transport_errors.MAX_RETRY_AFTER_SECONDS	No longer re-exported via _session or _core.
notebooklm._core.DEFAULT_* (timeouts, concurrency knobs)	notebooklm._runtime.config.DEFAULT_*
notebooklm._core.AUTH_ERROR_PATTERNS, notebooklm._core.is_auth_error	notebooklm._runtime.helpers
notebooklm._core.ERROR_INJECT_ENV_VAR	notebooklm._error_injection.ERROR_INJECT_ENV_VAR
notebooklm._core._SyntheticErrorTransport (class)	Removed	Synthetic-error substitution moved into notebooklm._middleware.error_injection.ErrorInjectionMiddleware. The env-var resolver (_get_error_injection_mode) and startup guard (_refuse_synthetic_error_outside_test_context) survive in notebooklm._error_injection.
notebooklm._core.AuthRefreshCoordinator	notebooklm._runtime.auth.AuthRefreshCoordinator	Class unchanged; only the home module moved.
notebooklm._core.TransportDrainTracker	notebooklm._transport_drain.TransportDrainTracker	Same.
notebooklm._core.ClientMetrics	notebooklm._client_metrics.ClientMetrics	Same.
notebooklm._core.ReqidCounter	notebooklm._reqid_counter.ReqidCounter	Same.
notebooklm._core.CookiePersistence	notebooklm._cookie_persistence.CookiePersistence	Same.
notebooklm._core.ClientLifecycle	notebooklm._runtime.lifecycle.ClientLifecycle	Same.
notebooklm._core.RpcExecutor	notebooklm._rpc_executor.RpcExecutor	Same.
notebooklm._core authed transport helpers	notebooklm._request_types + notebooklm._transport_errors + notebooklm._streaming_post	The interim authed-transport Adapter and catch-all helper module were retired; request types, transport errors, and streaming POST behavior now have separate owning modules.

New modules introduced by Tier 12 / 13

These modules did not exist before Tier 12 began:

Module	Purpose
notebooklm._session_contracts	Historical v0.5.0 home for AuthMetadata, Kernel, and the shared capability Protocols (RpcCaller, LoopGuard, OperationScopeProvider, AsyncWorkRuntime) added in the capability refactor (ADR-0013). Current home: notebooklm._runtime.contracts, exporting only Kernel, RpcCaller, and LoopGuard; AuthMetadata is local to _source/upload.py, OperationScopeProvider is local to _artifact/polling.py, and AsyncWorkRuntime was deleted.
notebooklm._kernel	Concrete Kernel transport core (owns the httpx.AsyncClient, exposes post / cookies / aclose). Located at root (src/notebooklm/_kernel.py), not nested.
notebooklm._middleware	Middleware chain primitives (Middleware Protocol, NextCall callable type, RpcRequest / RpcResponse envelope dataclasses, build_chain composer).
notebooklm._middleware_tracing	Tier 12 PR 12.3 — request tracing middleware.
notebooklm._middleware_metrics	Tier 12 PR 12.4 — metrics collection middleware.
notebooklm._middleware_drain	Tier 12 PR 12.5 — drain bookkeeping middleware.
notebooklm._middleware.error_injection	Test-only error-injection middleware.
notebooklm._middleware_retry	Tier 12 PR 12.7 — 429 / 5xx retry middleware.
notebooklm._middleware_auth_refresh	Tier 12 PR 12.8 — auth-refresh-on-401 middleware.
notebooklm._middleware_semaphore	Tier 12 PR 12.9 — global RPC concurrency cap.
notebooklm._chat.transport	Chat-domain consumer-side error mapping over the shared authed POST pipeline. Replaces the chat-side wrapper that previously lived on _core.rpc_call.
notebooklm._transport_errors	Terminal Kernel.post error mapping into transport exceptions consumed by retry/auth middleware.
notebooklm._request_types	Shared dataclasses + type aliases for authed-POST request construction: AuthSnapshot, BuildRequest, PostBody, and BuildRequestResult.
notebooklm._streaming_post	Size-capped streaming POST helper used by Kernel.post.

Deleted symbols and changed defaults

Symbol or default	Replacement / new behavior
notebooklm._core._SyntheticErrorTransport (deleted)	notebooklm._middleware.error_injection.ErrorInjectionMiddleware (chain-resident; mode is still resolved from NOTEBOOKLM_VCR_RECORD_ERRORS via _error_injection._get_error_injection_mode).
Strict-decode soft mode	Strict decoding is now the only mode. The legacy lenient opt-out was removed in v0.7.0; setting old lenient-decode environment toggles no longer restores legacy behavior. See ADR-0011.

Design intent

The capability refactor (ADR-0013) replaced the broad _session_contracts.Session Protocol that Tier 13 originally shipped. That Protocol had become a capability bag:

class Session(Protocol):
    auth: AuthMetadata
    kernel: Kernel
    async def rpc_call(...) -> Any: ...
    async def transport_post(...) -> httpx.Response: ...
    async def next_reqid(...) -> int: ...
    def assert_bound_loop(self) -> None: ...
    def operation_scope(self, label: str) -> AbstractAsyncContextManager[None]: ...
    def register_drain_hook(self, name, hook) -> None: ...

That shape violated the intent of ADR-0010: feature APIs should not depend on concrete Session internals. It also made dependencies hard to read — most features only needed logical RPC calls, while chat, uploads, and artifact polling needed narrower specialized runtime slices.

Design rules

The capability model is built on six rules:

1. Promote a capability to _runtime/contracts.py only when it is shared by more than one feature or service, except for Kernel, which is the typed transport surface of the concrete client-owned kernel.
2. Keep single-feature runtime needs local to the owning feature module.
3. Feature-facing Protocols must not advertise unrelated capabilities from a broad host object.
4. Prefer feature-owned collaborators over widening shared session contracts.
5. Remove old core vocabulary from touched feature APIs.
6. Do not use mixins for dependency expression. Use Protocols for required capabilities and collaborators / services for extracted behavior.

Shared capability Protocols

src/notebooklm/_session_contracts.py originally ended up containing only shared capability Protocols:

class RpcCaller(Protocol):
    async def rpc_call(
        self,
        method: RPCMethod,
        params: list[Any],
        source_path: str = "/",
        allow_null: bool = False,
        _is_retry: bool = False,
        *,
        disable_internal_retries: bool = False,
        operation_variant: str | None = None,
    ) -> Any: ...


class LoopGuard(Protocol):
    def assert_bound_loop(self) -> None: ...


class OperationScopeProvider(Protocol):
    def operation_scope(self, label: str) -> AbstractAsyncContextManager[None]: ...


class AsyncWorkRuntime(LoopGuard, OperationScopeProvider, Protocol):
    """Runtime support for feature-owned async work."""

Current code has tightened that further: _runtime/contracts.py exports Kernel, RpcCaller, and LoopGuard; OperationScopeProvider is local to _artifact/polling.py; AuthMetadata is local to _source/upload.py; AsyncWorkRuntime was removed.

The following were not globally promoted:

• auth, kernel, transport_post(...), next_reqid(...) — needed only by uploads and chat respectively. They live on the concrete Session and on feature-local runtimes that consume them, not on a shared Protocol.
• register_drain_hook(...) — kept on the concrete drain collaborator passed into ArtifactsAPI, since artifact polling is the only behavior that registers close-time feature cleanup.

Feature-local runtimes

For part of the migration, features that needed a specialised slice of runtime capability declared that slice as a Protocol in the feature's own module:

• ChatRuntime in _chat.py — RpcCaller + LoopGuard + transport_post + next_reqid. transport_post is consumed via chat_aware_authed_post(...) in _chat/transport.py.
• ArtifactsRuntime in _artifacts.py — RpcCaller + AsyncWorkRuntime + DrainHookRegistration. Artifact polling owns the only close-time cleanup hook in the codebase today.
• UploadRuntime in _source_upload.py — RpcCaller + OperationScopeProvider + LoopGuard. The upload pipeline also receives kernel and auth as constructor args, since uploads need upload-specific auth routing and live cookies — but those wires are explicit, not implicit through a god-object.

If a future feature ever needs the same local slice that another feature has carved out, the rule is: promote the Protocol to _runtime/contracts.py only when there is a real second consumer, not on speculation.

Those composite runtime Protocols were later removed. Current constructors take the direct collaborators they need by keyword: ArtifactsAPI(rpc=..., drain=..., lifecycle=...), ChatAPI(rpc=..., transport=..., reqid=..., loop_guard=...), and SourceUploadPipeline(rpc=..., drain=..., lifecycle=..., kernel=..., auth=...).

Note / mind-map service split

The capability refactor also separated note-row primitives from mind-map product behavior. Before the split, _mind_map.py mixed three responsibilities:

1. Backend row classification (note vs. mind-map vs. saved-chat).
2. Mind-map filtering and content extraction.
3. Saved-from-chat note encoding (chat citations + rich source passage anchors).

These were factored apart:

• _note_service.py owns the generic note row primitives: fetch_note_rows, classify_row (over a private NoteRowKind enum), extract_content, create_note, update_note, delete_note. Mind maps are stored in the same note-row collection that NotebookLM returns from GET_NOTES_AND_MIND_MAPS, so note-row classification is the right place to recognise them.
• _mind_map.py kept the mind-map boundary as NoteBackedMindMapService — list_mind_maps, extract_content, delete_mind_map — delegating to NoteService for the underlying row operations. NotesAPI.list_mind_maps(...) and NotesAPI.delete_mind_map(...) forward through this service, preserving their public signatures.
• Saved-from-chat note encoding moved out of _mind_map.py and into _chat/notes.py, where ChatAPI.save_answer_as_note(...) owns the workflow. NotesAPI.create_from_chat(...) was a deprecated forwarder during the migration window and was removed in v0.7.0.

NoteRowKind stays private — it is an internal classification of rows returned by the undocumented GET_NOTES_AND_MIND_MAPS RPC, not a public API type.

Constructor and naming rules

Feature APIs adopted consistent dependency-naming conventions:

• Pure-RPC features store self._rpc. Constructor parameter is named rpc: RpcCaller.
• Features with multiple runtime needs take direct keyword-only collaborators and store only the fields they use.
• The old self._core vocabulary was removed from every touched feature module.
• Extra collaborators are keyword-only:

SourcesAPI(rpc, *, uploader=source_uploader)
NotebooksAPI(rpc, *, sources_api=sources)
ChatAPI(rpc=rpc, transport=transport, reqid=reqid, loop_guard=lifecycle,
        notebooks=notebooks)
ArtifactsAPI(rpc=rpc, drain=drain, lifecycle=lifecycle,
             notebooks=notebooks, mind_maps=mind_maps,
             note_service=note_service)
NotesAPI(*, notes=note_service, mind_maps=mind_maps)
MindMapsAPI(rpc=rpc, mind_maps=mind_maps, artifacts=artifacts,
            notebooks=notebooks)
LabelsAPI(rpc, list_sources=sources.list)

Compatibility aliases and fallback constructors that read missing collaborators off a broad session were removed — every feature now requires its dependencies explicitly. _client_assembly.py::_assemble_client is the single wiring root that hands them out.

How it landed

The refactor arc shipped in four phases against the refactor-completion-plan series (PRs in the #84x–#92x range). Detailed phase notes live in the merged PR descriptions; the high-level shape was:

• Phase 1 — additive contracts. New capability Protocols (RpcCaller, LoopGuard, OperationScopeProvider, AsyncWorkRuntime) were added to _session_contracts.py alongside the existing broad Session Protocol. Nothing was removed. Build stayed green.
• Phase 2 — pure-RPC feature retyping. NotebooksAPI, ResearchAPI, SettingsAPI, SharingAPI were retyped to RpcCaller, with self._core renamed to self._rpc. The FakeSession test fixture shrunk in the same commits to match the narrower contract.
• Phase 3 — feature-local runtimes. ChatRuntime, ArtifactsRuntime, and UploadRuntime were introduced in their owning modules. ChatAPI, ArtifactsAPI, SourcesAPI, and SourceUploadPipeline migrated to the new local Protocols. Compatibility fallbacks (the drain_hooks=None -> session fallback in ArtifactsAPI, the session.kernel / session.auth / record_upload_queue_wait fallbacks in SourcesAPI, the core= alias on ChatAPI) were removed in the same commits that updated their direct-construction test sites.
• Phase 4 — note / mind-map split + capability cleanup. NoteService and NoteBackedMindMapService were introduced, artifact generation and download paths were rewired through them, ChatAPI.save_answer_as_note(...) was added, and NotesAPI.create_from_chat(...) was converted to a deprecated forwarder (later removed in v0.7.0). The module-level _mind_map wrappers were removed. The broad Session Protocol was deleted from _session_contracts.py, along with the broad FakeSession defaults shape and the broad-Protocol test pin. The _core.py compatibility shim and the NotebookLMClient._core attribute alias were both removed in this phase (#889).

Implementation tactics — line-number-specific constructor changes, the ordering rules for compatibility-fallback removals, the _source_upload.py local RpcCaller → RpcCallback rename to avoid name collision with the new shared Protocol — are recorded in the merged PR descriptions, not in this document. The post-refactor runtime shape is canonicalized in docs/architecture.md.

See also

• docs/architecture.md — current runtime shape, capability-protocol model, and dispatch path.
• ADR-0010 — Session / Kernel split — the design driver for Tier 13. Superseded by ADR-0013.
• ADR-0011 — Schema validation policy — the strict-decode default flip.
• ADR-0012 — Implementation-surface convention.
• ADR-0013 — Composable session capabilities — the ratifying decision for the capability model described here.
• docs/stability.md — public stability contract.
• docs/python-api.md — canonical public API reference.