wehub-resource-sync
57 lines
4.3 KiB
Markdown
57 lines
4.3 KiB
Markdown
# Content Processing: Chunking, Embedding, Context & Encryption
|
|
|
|
Design notes for the utilities in `open_notebook/utils/` that turn raw content into searchable, LLM-consumable data. These are cross-cutting: sources, notes and insights all flow through them.
|
|
|
|
## Chunking (`utils/chunking.py`)
|
|
|
|
Content is split with content-type-aware LangChain splitters (`HTMLHeaderTextSplitter`, `MarkdownHeaderTextSplitter`, `RecursiveCharacterTextSplitter`). Content type detection uses the file extension first; heuristics can override a PLAIN extension when confidence ≥ 0.8. Oversized chunks from the HTML/Markdown splitters get a secondary split.
|
|
|
|
**Why the 400-token default.** `OPEN_NOTEBOOK_CHUNK_SIZE` defaults to 400 tokens — ~20% below the 512-token ceiling of BERT-family embedders (e.g. `mxbai-embed-large`). The buffer absorbs three error sources: tokenizer mismatch (we measure with `o200k_base`, the embedder tokenizes with WordPiece), splitter overshoot, and special tokens. For embedders with large windows (OpenAI `text-embedding-3` family: 8191 tokens) raise it, e.g.:
|
|
|
|
```bash
|
|
export OPEN_NOTEBOOK_CHUNK_SIZE=1500
|
|
export OPEN_NOTEBOOK_CHUNK_OVERLAP=150
|
|
```
|
|
|
|
`OPEN_NOTEBOOK_CHUNK_OVERLAP` defaults to 15% of chunk size. Both are **token-based** (not characters), minimum chunk size 100, and require an app restart to take effect.
|
|
|
|
## Embedding (`utils/embedding.py`)
|
|
|
|
- `generate_embedding(text)` — unified entry point: short text (≤ chunk size) embeds directly; long text is chunked, each chunk embedded, and the results combined via **mean pooling** (normalize each → mean → normalize result, numpy).
|
|
- `generate_embeddings(texts)` — batch path used by `embed_source_command`: batches of 50 with per-batch retry, to stay under provider payload limits.
|
|
- Empty/whitespace-only input raises `ValueError` — which background commands treat as a permanent (non-retried) failure by design.
|
|
- The embedding model comes from `model_manager` (see [credentials.md](credentials.md) for how provider config is resolved).
|
|
|
|
**Who triggers embedding** (see also the domain rules in `open_notebook/AGENTS.md`):
|
|
|
|
| Content | Trigger |
|
|
|---|---|
|
|
| Note | `Note.save()` auto-submits `embed_note` |
|
|
| Insight | `create_insight_command` submits `embed_insight` |
|
|
| Source | explicit `source.vectorize()` → `embed_source` (NOT automatic on save) |
|
|
| Everything | `rebuild_embeddings_command` fans out individual jobs |
|
|
|
|
All embedding is fire-and-forget through the surreal-commands worker — nothing embeds if the worker isn't running.
|
|
|
|
## Context building (`utils/context_builder.py`)
|
|
|
|
The single implementation behind both context consumers:
|
|
|
|
- `build_notebook_context()` backs `POST /api/chat/context` (chat panel + podcast generation): it assembles source/note contexts from the inclusion config, whose status strings are matched textually ("not in" skips, "insights" → short context, "full content" → long context). Without a config, every source and note is included with its short context. Per-item failures are logged and skipped.
|
|
- `build_source_context()` backs the source-chat graph: one source's short context plus its insights, truncated to a token budget by dropping insights (last-fetched first).
|
|
- Every call re-fetches — there is no cache layer.
|
|
- Token counting uses `o200k_base` via tiktoken and is an estimate (±5-10% vs. the actual model); `token_count()` falls back to a coarse estimate if tiktoken is unavailable.
|
|
|
|
## Encryption (`utils/encryption.py`) {#encryption}
|
|
|
|
Field-level encryption for sensitive values (API keys) stored in the database, using Fernet (AES-128-CBC + HMAC-SHA256).
|
|
|
|
- Key source: `OPEN_NOTEBOOK_ENCRYPTION_KEY_FILE` (Docker secrets) → `OPEN_NOTEBOOK_ENCRYPTION_KEY`. **No default** — credential storage is unavailable until the key is set.
|
|
- Any string works as key: it's derived to a Fernet key via SHA-256, lazily on first use.
|
|
- Decryption falls back gracefully: an `InvalidToken` (legacy unencrypted data) returns the original value, so pre-encryption databases keep working.
|
|
- Key rotation is **not implemented** — changing the key orphans previously encrypted values.
|
|
|
|
## Text utilities (`utils/text_utils.py`)
|
|
|
|
`clean_thinking_content()` strips `<think>…</think>` blocks from model output (extended-thinking models); used in every graph that consumes LLM responses. It handles malformed output (missing opening tag) and bypasses extraction for content > 100KB for performance.
|