chore: import upstream snapshot with attribution

2026-07-13 13:28:29 +08:00
commit fed8b2eed7
1531 changed files with 1107494 additions and 0 deletions
@@ -0,0 +1,114 @@
+---
+title: GraphRAG — Knowledge-Graph Retrieval
+description: Build a knowledge graph from a source at ingest time and retrieve over it with Personalized PageRank. Covers requirements, enabling, configuration, and the graph view.
+---
+
+import { Callout } from 'nextra/components'
+import Image from 'next/image'
+
+# GraphRAG
+
+GraphRAG augments classic vector retrieval with a **knowledge graph**. During ingestion DocsGPT uses an LLM to extract entities and the relationships between them from a source's chunks, and stores them as a graph alongside the vectors. At query time, a graph retriever uses Personalized PageRank (PPR) to walk that graph from the entities mentioned in your question, surfacing connected context that pure similarity search can miss — useful for multi-hop questions and queries that span related concepts.
+
+<Callout type="warning" emoji="⚠️">
+GraphRAG is **flag-gated** and currently **pgvector-only**. It is available only when both `GRAPHRAG_ENABLED=true` **and** `VECTOR_STORE=pgvector`. On any other vector store the enable action is rejected.
+</Callout>
+
+## Requirements
+
+- A PostgreSQL database with the `pgvector` extension (`VECTOR_STORE=pgvector`). See [PostgreSQL for User Data](/Deploying/Postgres-Migration).
+- `GRAPHRAG_ENABLED=true` in your environment.
+- An LLM configured for extraction (GraphRAG reuses your instance default model unless you override it).
+
+```env
+GRAPHRAG_ENABLED=true
+VECTOR_STORE=pgvector
+```
+
+The graph tables live in the same pgvector database as your embeddings and are sized to the embedding dimension. If you change embedding models you must re-ingest and re-extract (see [Embeddings](/Models/embeddings#important-embedding-dimensions-must-stay-consistent)).
+
+## How it works
+
+1. **Choose GraphRAG** for the source — either at upload time, or by enabling it on an existing source (see below). This sets the source's config to `graphrag` mode.
+2. **Extraction** runs over the source's chunks. For each chunk, the LLM extracts entities and relations, which are written into per-source graph tables. Extraction is durable and resumable via a checkpoint, so it survives restarts and re-runs from scratch each time you re-enable it.
+3. **Query.** Questions against the source are routed to the graph retriever, which runs Personalized PageRank from the query's entities to gather related context.
+
+<Callout type="info" emoji="ℹ️">
+If a source has no graph yet (extraction still running or failed), the graph retriever **falls back to classic vector retrieval** for that source — answers keep working, they just don't use the graph until it is ready.
+</Callout>
+
+## Enabling GraphRAG
+
+### At upload time (recommended)
+
+When you upload a new document, open **Advanced settings** and set **Retriever** to **GraphRAG** (the same dropdown also offers **Hybrid**). The source is created in `graphrag` mode and extraction is enqueued as part of ingestion — no extra step.
+
+<Image
+  src="/graph-rag-settings-before-upload.png"
+  alt="Upload dialog advanced settings showing the Retriever dropdown with Classic, Hybrid, and GraphRAG options"
+  width={661}
+  height={945}
+/>
+
+These are the same [per-source retrieval settings](/Sources/Per-source-configuration) you can change later — choosing the retriever up front just avoids a re-ingest.
+
+### On an existing source
+
+To turn an already-ingested source into a GraphRAG source, use the **Enable GraphRAG** action on the source (it shows a status badge while extraction runs), or call the API:
+
+```bash
+curl -X POST https://your-docsgpt/api/sources/<source_id>/graphrag/enable \
+  -H "Authorization: Bearer <token>"
+```
+
+The response returns a `task_id` for the extraction job:
+
+```json
+{ "success": true, "task_id": "..." }
+```
+
+Notes:
+
+- Requires write access to the source (owner or team `editor`).
+- Returns `400` if GraphRAG isn't available on the workspace (wrong vector store or flag off).
+- Re-running the action rebuilds the graph from scratch rather than no-opping against an existing one.
+- You cannot switch a source to `graphrag` through the [config PATCH endpoint](/Sources/Per-source-configuration#editing-the-config-via-api) — use the upload-time selector or this dedicated endpoint.
+
+## Configuration
+
+Instance-wide settings (see [App Configuration](/Deploying/DocsGPT-Settings)):
+
+| Setting | Default | Description |
+| --- | --- | --- |
+| `GRAPHRAG_ENABLED` | `false` | Master switch for the feature. |
+| `GRAPHRAG_EXTRACTION_MODEL` | `null` | Model used for extraction. `null` reuses the instance default model. |
+| `GRAPHRAG_MAX_CHUNKS_FOR_EXTRACTION` | `2000` | Hard cap on how many chunks are extracted per source (cost control). |
+
+Per-source extraction knobs live under the source config's `graph` object and override the instance defaults:
+
+| Field | Default | Description |
+| --- | --- | --- |
+| `extraction_model` | `null` | Override the extraction model for this source. |
+| `max_chunks` | `null` | Override the chunk cap; `null` falls back to `GRAPHRAG_MAX_CHUNKS_FOR_EXTRACTION`. |
+| `gleanings` | `0` | Extra extraction passes per chunk to catch entities missed on the first pass. Off by default (each pass costs additional LLM calls). |
+
+<Callout type="warning" emoji="⚠️">
+Graph extraction makes an LLM call per chunk (more if `gleanings > 0`), so it has a real token cost. The cost is attributed to token usage under a `graph_extraction` tag, and the `max_chunks` cap bounds it.
+</Callout>
+
+## Visualizing the graph
+
+GraphRAG sources expose a **graph view** in the UI — an interactive network of the extracted entities and relationships. It is backed by two read endpoints:
+
+```text
+GET /api/sources/<source_id>/graph                    # bounded {nodes, edges} overview
+GET /api/sources/<source_id>/graph/node/<node_id>     # one node and its neighbors
+```
+
+The overview is bounded to a default node limit to keep large graphs responsive.
+
+## Related
+
+- [Per-Source Configuration](/Sources/Per-source-configuration) — the config object GraphRAG plugs into.
+- [PostgreSQL for User Data](/Deploying/Postgres-Migration) — required pgvector setup.
+- [Embeddings](/Models/embeddings) — embedding-dimension constraints that also apply to the graph tables.
@@ -0,0 +1,177 @@
+---
+title: Per-Source Configuration (Chunking & Retrieval)
+description: Tune how each knowledge source is chunked at ingest time and retrieved at query time — chunking strategy, retriever, exposure, top-k, pre-screening and more.
+---
+
+import { Callout } from 'nextra/components'
+import Image from 'next/image'
+
+# Per-Source Configuration
+
+Every source in DocsGPT carries its own **behavior contract** — a small config object that controls how that source is *chunked* when it is ingested and how it is *retrieved* when you ask a question. This lets you tune each source independently: a large reference manual can use a different chunking strategy and retriever than a short FAQ.
+
+You edit this config from a source's settings in the UI (shown below), or through the API. The same options are also available in **Advanced settings** when you first upload a document.
+
+<Image
+  src="/sources-settings-screen.png"
+  alt="Source settings panel showing Retrieval options (retriever, top-k, score threshold, rephrase, exposure, prescreen) and Chunking options (strategy, max/min tokens, duplicate headers)"
+  width={633}
+  height={862}
+/>
+
+<Callout type="info" emoji="ℹ️">
+Per-source retrieval is enabled by default. Operators can turn it off instance-wide with `PER_SOURCE_RETRIEVAL_ENABLED=false`, in which case all sources fall back to the classic retriever regardless of their stored config.
+</Callout>
+
+## Two kinds of settings: live vs. bake-time
+
+The config has two groups of settings that differ in *when* they take effect:
+
+| Group | When it applies | Re-ingest needed? |
+| --- | --- | --- |
+| **Retrieval** (`retrieval.*`) | Query time — applied live on the next question | No |
+| **Chunking** (`chunking.*`) | Ingest time — baked into the stored chunks | **Yes** |
+
+Changing a retrieval setting takes effect immediately. Changing a chunking setting only affects documents ingested *after* the change, so you must re-ingest the source to apply it to existing content. The API response includes a `requires_reingest` flag to make this explicit.
+
+## Chunking configuration
+
+Chunking decides how a document is split into the pieces that get embedded and stored.
+
+```json
+{
+  "chunking": {
+    "strategy": "classic_chunk",
+    "max_tokens": 1250,
+    "min_tokens": 150,
+    "duplicate_headers": false
+  }
+}
+```
+
+| Field | Default | Description |
+| --- | --- | --- |
+| `strategy` | `classic_chunk` | Which chunking algorithm to use (see below). |
+| `max_tokens` | `1250` | Upper bound on chunk size in tokens. |
+| `min_tokens` | `150` | Lower bound; small fragments are merged up to this size. |
+| `duplicate_headers` | `false` | Repeat section headers into each child chunk for context. |
+
+### Available chunking strategies
+
+| Strategy | Behavior |
+| --- | --- |
+| `classic_chunk` | The default token-window splitter. An empty config reproduces DocsGPT's historical chunking byte-for-byte. |
+| `recursive` | Recursive character/token splitter that tries to break on natural boundaries (paragraphs, sentences). |
+| `markdown` | Splits along Markdown structure (headings, sections) — good for docs and wikis. |
+| `parent_child` | Embeds small **child** chunks for precise matching but carries a larger **parent** window in metadata, so the model still sees surrounding context. |
+| `semantic` | Embeds sentences and splits where meaning shifts (at the 95th-percentile cosine-distance gap between adjacent sentences), falling back to `recursive` on failure. Produces topically coherent chunks at the cost of extra embedding calls during ingest. |
+
+<Callout type="warning" emoji="⚠️">
+Chunking is bake-time. After changing `strategy`, `max_tokens`, `min_tokens`, or `duplicate_headers`, re-ingest the source so existing chunks are rebuilt.
+</Callout>
+
+## Retrieval configuration
+
+Retrieval decides which chunks are pulled in to answer a question. These settings apply live.
+
+```json
+{
+  "retrieval": {
+    "retriever": "classic",
+    "exposure": "prefetch",
+    "chunks": 2,
+    "score_threshold": null,
+    "rephrase_query": true,
+    "prescreen": null
+  }
+}
+```
+
+| Field | Default | Description |
+| --- | --- | --- |
+| `retriever` | `classic` | Retrieval strategy: `classic`, `hybrid`, or `graphrag`. |
+| `exposure` | `prefetch` | How retrieved context reaches the model: `prefetch` or `agentic_tool` (see below). |
+| `chunks` | `2` | Final number of chunks (top-k) returned to the answer. Range 1–500. |
+| `score_threshold` | `null` | Minimum similarity score. Honored by pgvector and MongoDB Atlas; other stores ignore it. |
+| `rephrase_query` | `true` | Whether to run a query-rephrasing side-call before retrieval. |
+| `prescreen` | `null` | Optional LLM relevance filter (see below). `null` = off. |
+
+### Retrievers
+
+- **`classic`** — Vector similarity search. The default and a safe choice for any vector store.
+- **`hybrid`** — Fuses vector search with full-text keyword search using Reciprocal Rank Fusion, which improves recall for exact terms, codes, and names that pure vector search can miss.
+- **`graphrag`** — Knowledge-graph retrieval. Set indirectly when you enable GraphRAG on a source. See [GraphRAG](/Sources/GraphRAG).
+
+<Callout type="warning" emoji="⚠️">
+Keyword search for the **hybrid** retriever is currently implemented only for the **pgvector** vector store. On other stores (FAISS, Qdrant, Milvus, etc.) the keyword half returns nothing, so `hybrid` quietly behaves like `classic` (vector-only).
+</Callout>
+
+Operators can restrict which retrievers are usable instance-wide with the `RETRIEVERS_ENABLED` setting; a per-source `retriever` value must be within that allow-list.
+
+### Exposure: prefetch vs. agentic tool
+
+`exposure` controls *how* a source's content is delivered to the model:
+
+- **`prefetch`** (default) — DocsGPT retrieves the top chunks up front and injects them into the prompt before the model answers. Best for focused Q&A over a source.
+- **`agentic_tool`** — The source is exposed to the model as a search tool it can call on demand, deciding when and what to look up (browse-as-you-go) rather than receiving a bulk prefetch. This is the default exposure for [Wiki sources](/Sources/Wiki-sources).
+
+### Pre-screening (LLM relevance filter)
+
+Pre-screening adds an optional map-reduce step between retrieval and answering: a base retriever fetches a wider set of candidates, an LLM screens them in batches, and only the most relevant survivors are passed to the answer. It improves precision on noisy sources at the cost of extra query-time LLM calls, so it is **off by default**.
+
+```json
+{
+  "retrieval": {
+    "chunks": 8,
+    "prescreen": {
+      "candidate_k": 40,
+      "batch_size": 10,
+      "max_keep": 8,
+      "model": null
+    }
+  }
+}
+```
+
+| Field | Default | Description |
+| --- | --- | --- |
+| `candidate_k` | `40` | Candidates fetched before screening. Must be `>= chunks`. |
+| `batch_size` | `10` | Candidates screened per LLM call. |
+| `max_keep` | `8` | Survivors kept after screening. Must be `<= candidate_k`. |
+| `model` | `null` | Model used for screening. `null` reuses the request's resolved model. |
+
+## Editing the config via API
+
+The config is edited with a `PATCH` to the source's config endpoint:
+
+```bash
+curl -X PATCH https://your-docsgpt/api/sources/<source_id>/config \
+  -H "Authorization: Bearer <token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "retrieval": { "retriever": "hybrid", "chunks": 4 },
+    "chunking":  { "strategy": "semantic" }
+  }'
+```
+
+The response echoes the stored config and a `requires_reingest` flag:
+
+```json
+{
+  "success": true,
+  "config": { "...": "..." },
+  "requires_reingest": true
+}
+```
+
+Notes:
+
+- Invalid values are rejected with `400` (strict validation on write).
+- The `kind` field (classic / wiki / graphrag) cannot be changed through this endpoint — converting a source to a [Wiki](/Sources/Wiki-sources) or enabling [GraphRAG](/Sources/GraphRAG) uses dedicated endpoints.
+- Editing requires ownership of the source or a team `editor` grant; viewers receive `403`.
+
+## Related
+
+- [GraphRAG](/Sources/GraphRAG) — knowledge-graph retrieval for a source.
+- [Wiki Sources](/Sources/Wiki-sources) — LLM-editable living documentation.
+- [Embeddings](/Models/embeddings) — the embedding model used during ingest and retrieval.
@@ -0,0 +1,96 @@
+---
+title: Wiki Sources — Living, LLM-Editable Documentation
+description: Create a knowledge source that the agent can read and write — a living wiki it keeps up to date, with human edits, provenance stamps, and version safety.
+---
+
+import { Callout } from 'nextra/components'
+
+# Wiki Sources
+
+A **wiki source** is a knowledge source that the agent can both read *and* write. Instead of being a fixed set of ingested files, a wiki is a small set of Markdown pages that the LLM edits over time — recording what it learns, correcting stale information, and building living documentation. Humans can edit the same pages directly, and every change is stamped with who made it.
+
+Unlike a classic source, a wiki is **team-scoped, not per-user**: it is shared and edited at the source level, so a whole team works against the same living document.
+
+## How a wiki differs from a classic source
+
+| | Classic source | Wiki source |
+| --- | --- | --- |
+| Content | Ingested files, read-only | Markdown pages, read **and** write |
+| Who edits | You (re-upload to change) | The agent and humans |
+| Default exposure | `prefetch` (chunks injected up front) | `agentic_tool` (the agent browses pages on demand) |
+| Searchability | Embedded at ingest | Re-embedded automatically on every edit |
+| Scope | Per owner | Team-shareable, edited at source scope |
+
+Because a wiki defaults to the `agentic_tool` [exposure](/Sources/Per-source-configuration#exposure-prefetch-vs-agentic-tool), the agent navigates it as a tool — opening, searching, and editing pages as needed — rather than receiving a bulk prefetch.
+
+## How the agent edits a wiki
+
+The agent edits a wiki through an internal **Wiki tool** that is automatically scoped to one wiki source. It supports a small, edit-safe action surface:
+
+- **view** a page,
+- **create** or overwrite a page,
+- **str_replace** an exact, unique string,
+- **insert** at a line,
+- **delete** a page,
+- **rename** a page.
+
+Two safety properties matter:
+
+- **Provenance stamps.** Every page records whether its last change came from a human or the agent, so edits are traceable.
+- **Optimistic versioning.** Edits carry an expected version; if a page changed underneath, the edit is rejected rather than silently clobbering a concurrent change. String replacements must match exactly and uniquely.
+
+After any edit, the affected page is **re-embedded asynchronously** so the wiki stays searchable and the new content is immediately retrievable.
+
+<Callout type="info" emoji="ℹ️">
+Each wiki page is capped at 1 MB (1,000,000 bytes). Pages are addressed by path (the home page is `/index.md`).
+</Callout>
+
+## Creating a wiki
+
+In the UI, choose **New wiki source** when adding a source. From the API:
+
+```bash
+curl -X POST https://your-docsgpt/api/sources/wiki \
+  -H "Authorization: Bearer <token>" \
+  -H "Content-Type: application/json" \
+  -d '{ "name": "Team Handbook", "initial_content": "# Team Handbook\n\nWelcome." }'
+```
+
+- `name` (required) — the wiki source name.
+- `initial_content` (optional) — Markdown that seeds the home page `/index.md` and triggers its first re-embed.
+
+No ingestion task runs for a wiki; pages are authored directly. The response returns the new `source_id`.
+
+## Converting an existing source into a wiki
+
+You can turn an already-ingested source into a wiki so the agent can start maintaining it:
+
+```bash
+curl -X POST https://your-docsgpt/api/sources/<source_id>/wiki/convert \
+  -H "Authorization: Bearer <token>"
+```
+
+- A **blank** source is enabled inline (no task) and immediately becomes a wiki.
+- A source **with files** runs a conversion task that reassembles its existing chunks into wiki pages; poll the returned task for a per-file summary.
+- Conversion is rejected with `409` if the source is still ingesting — wait for it to finish first.
+
+<Callout type="warning" emoji="⚠️">
+Switching a source to (or from) wiki mode goes only through `POST /api/sources/<id>/wiki/convert`. It cannot be done through the [config PATCH endpoint](/Sources/Per-source-configuration#editing-the-config-via-api).
+</Callout>
+
+## Reading and editing pages directly
+
+Humans can read and edit wiki pages through the API (and the wiki viewer in the UI):
+
+```text
+GET  /api/sources/<source_id>/wiki/pages          # list pages
+GET  /api/sources/<source_id>/wiki/page?path=...  # fetch one page fresh
+PUT  /api/sources/<source_id>/wiki/page           # create or overwrite a page (human edit)
+```
+
+Human edits are stamped with `human` provenance and trigger the same re-embed as agent edits. Read access follows source sharing (owner or anyone the source is shared with); writing requires owner or team `editor` access.
+
+## Related
+
+- [Per-Source Configuration](/Sources/Per-source-configuration) — exposure and retrieval settings a wiki uses.
+- [Access Control & Teams](/Deploying/Access-Control) — sharing a wiki with a team.
@@ -0,0 +1,14 @@
+export default {
+  "Per-source-configuration": {
+    "title": "🎛️ Per-Source Configuration",
+    "href": "/Sources/Per-source-configuration"
+  },
+  "GraphRAG": {
+    "title": "🕸️ GraphRAG",
+    "href": "/Sources/GraphRAG"
+  },
+  "Wiki-sources": {
+    "title": "📖 Wiki Sources",
+    "href": "/Sources/Wiki-sources"
+  }
+}