chore: import upstream snapshot with attribution

docs/AsymmetricEmbedding.md

@@ -0,0 +1,179 @@
+# Asymmetric Embedding Configuration
+
+LightRAG keeps embedding behavior symmetric by default. Query/document asymmetric
+embedding is enabled only when `EMBEDDING_ASYMMETRIC=true` is explicitly set.
+
+This avoids accidental retrieval changes when prefix variables are present in an
+environment but the user did not intentionally enable asymmetric embeddings.
+
+Before enabling asymmetric embeddings for any model, check the model's current
+model card or provider documentation. Do not infer the right behavior from the
+API binding alone: an `openai`-compatible endpoint can serve instruction-free
+models, prefix-based models, or provider-specific models behind the same API
+shape.
+
+## Reindexing Requirement
+
+Changing asymmetric embedding settings changes the vectors produced for stored
+documents and for future queries. After enabling, disabling, or changing any of
+these settings, clear the existing LightRAG data for the workspace and re-index
+the source files:
+
+- `EMBEDDING_ASYMMETRIC`
+- `EMBEDDING_QUERY_PREFIX`
+- `EMBEDDING_DOCUMENT_PREFIX`
+- Provider task behavior such as Jina `task`, Gemini `task_type`, or VoyageAI
+  `input_type`
+
+Do not reuse an existing vector store across asymmetric embedding configuration
+changes. Mixing vectors generated with different query/document behavior can
+make retrieval quality unpredictable.
+
+## Binding Types
+
+LightRAG distinguishes two asymmetric embedding styles:
+
+| Style | Bindings | How asymmetric behavior is applied |
+| --- | --- | --- |
+| Provider task parameters | `jina`, `gemini`, `voyageai` | LightRAG passes query/document context to the provider-specific `task`, `task_type`, or `input_type` parameter. |
+| Text task prefixes | `openai`, `azure_openai`, `ollama` | LightRAG prepends configured text prefixes before calling the embedding API. Use this only when the model card explicitly requires separate query/document prefixes. |
+
+Other server embedding bindings do not currently support
+`EMBEDDING_ASYMMETRIC=true`.
+
+## Default: Symmetric Embeddings
+
+When `EMBEDDING_ASYMMETRIC` is unset, LightRAG does not enable asymmetric
+embedding behavior, even if prefix variables exist:
+
+```env
+# EMBEDDING_ASYMMETRIC is unset
+# EMBEDDING_QUERY_PREFIX="search_query: "
+# EMBEDDING_DOCUMENT_PREFIX="search_document: "
+```
+
+The prefixes are ignored and a warning is logged.
+
+The same is true when the flag is explicitly false:
+
+```env
+EMBEDDING_ASYMMETRIC=false
+```
+
+## Instruction-Free Models: Keep Symmetric
+
+Some embedding models are instruction-free, sometimes described as using
+implicit intent. They are trained to handle query/document matching from the raw
+text itself and do not require query/document prefixes or provider task
+parameters. For these models, do not set `EMBEDDING_ASYMMETRIC=true`; leave it
+unset or set it to `false`, and do not configure `EMBEDDING_QUERY_PREFIX` or
+`EMBEDDING_DOCUMENT_PREFIX`.
+
+Common examples that should normally stay in symmetric mode:
+
+| Model family | Example model IDs | Notes |
+| --- | --- | --- |
+| BGE-M3 | `BAAI/bge-m3` | Use plain text input. Do not add `search_query:` / `search_document:` unless the specific serving wrapper's model card says otherwise. |
+| OpenAI Text Embedding 3 | `text-embedding-3-small`, `text-embedding-3-large` | The OpenAI embeddings API uses text input plus the model name; it does not expose a query/document task parameter. |
+| Mistral Embed | `mistral-embed` | Use the provider's plain embedding input. Do not invent task prefixes. |
+| Alibaba GTE base models | `gte-large`, `gte-large-zh` | Base GTE models use plain text for normal retrieval. This does not apply to newer `instruct` variants such as `gte-Qwen2-1.5B-instruct`; check that model card. |
+| Jina Embeddings v2 | `jina-embeddings-v2-base-en`, `jina-embeddings-v2-base-zh` | Jina v2 is plain-text input. Jina v3/v4 are different and use the `task` parameter for retrieval tasks. |
+
+If a model is instruction-free, enabling LightRAG's asymmetric mode can make the
+input different from what the model was trained or documented to expect. That can
+reduce retrieval quality even though the server starts successfully.
+
+## Provider Task Parameter Bindings
+
+Use this mode for providers that expose separate query/document embedding tasks.
+Do not configure prefix variables for these bindings.
+
+Jina example:
+
+```env
+EMBEDDING_BINDING=jina
+EMBEDDING_ASYMMETRIC=true
+EMBEDDING_MODEL=jina-embeddings-v4
+```
+
+Gemini example:
+
+```env
+EMBEDDING_BINDING=gemini
+EMBEDDING_ASYMMETRIC=true
+EMBEDDING_MODEL=gemini-embedding-001
+```
+
+VoyageAI example:
+
+```env
+EMBEDDING_BINDING=voyageai
+EMBEDDING_ASYMMETRIC=true
+EMBEDDING_MODEL=voyage-3
+```
+
+If `EMBEDDING_QUERY_PREFIX` or `EMBEDDING_DOCUMENT_PREFIX` is also configured
+for these bindings, LightRAG logs a warning and ignores the prefixes.
+
+## Text Task Prefix Bindings
+
+Use this mode for embedding models that expect task instructions in the input
+text, such as models whose card documents prefixes like `search_query:`,
+`search_document:`, `query:`, or `passage:`. Do not enable this mode just
+because the model is served through `openai`, `azure_openai`, or `ollama`.
+
+Both prefix variables must be explicitly configured:
+
+```env
+EMBEDDING_ASYMMETRIC=true
+EMBEDDING_QUERY_PREFIX="search_query: "
+EMBEDDING_DOCUMENT_PREFIX="search_document: "
+```
+
+If one side should intentionally have no prefix, use the sentinel `NO_PREFIX`:
+
+```env
+EMBEDDING_ASYMMETRIC=true
+EMBEDDING_QUERY_PREFIX="search_query: "
+EMBEDDING_DOCUMENT_PREFIX=NO_PREFIX
+```
+
+`NO_PREFIX` is converted to an empty string internally. It is different from an
+unset variable: it means the side was reviewed and intentionally left without a
+prefix.
+
+At least one side must have a non-empty prefix. This is invalid:
+
+```env
+EMBEDDING_ASYMMETRIC=true
+EMBEDDING_QUERY_PREFIX=NO_PREFIX
+EMBEDDING_DOCUMENT_PREFIX=NO_PREFIX
+```
+
+## Invalid Empty Prefixes
+
+Do not use an empty environment value for an intentional empty prefix:
+
+```env
+EMBEDDING_DOCUMENT_PREFIX=
+```
+
+Use `NO_PREFIX` instead. Empty values are rejected because shell, `.env`, and
+Docker Compose handling can make empty strings indistinguishable from accidental
+missing configuration.
+
+## Validation Summary
+
+| Configuration | Result |
+| --- | --- |
+| `EMBEDDING_ASYMMETRIC` unset | Symmetric mode; prefixes ignored with a warning. |
+| `EMBEDDING_ASYMMETRIC=false` | Symmetric mode; prefixes ignored with a warning. |
+| Instruction-free model such as `BAAI/bge-m3`, `text-embedding-3-small`, `mistral-embed`, base GTE, or Jina v2 | Keep symmetric mode; do not configure prefixes or provider tasks unless the model card says to. |
+| `EMBEDDING_ASYMMETRIC=true` with `jina`/`gemini`/`voyageai` | Provider task mode; prefixes ignored with a warning. |
+| `EMBEDDING_ASYMMETRIC=true` with `openai`/`azure_openai`/`ollama` and both prefix variables configured | Prefix mode. |
+| Prefix mode with a missing prefix variable | Startup error; use a real prefix or `NO_PREFIX`. |
+| Prefix mode with both sides `NO_PREFIX` | Startup error; no asymmetric behavior would occur. |
+| Prefix variable set to an empty value | Startup error; use `NO_PREFIX`. |
+
+Any valid change from one asymmetric embedding configuration to another still
+requires clearing the workspace data and re-indexing the source files.

docs/DockerDeployment.md

@@ -0,0 +1,357 @@
+# LightRAG Docker Deployment
+
+A lightweight Knowledge Graph Retrieval-Augmented Generation system with multiple LLM backend support.
+
+## 🚀 Preparation
+
+### Clone the repository:
+
+```bash
+# Linux/MacOS
+git clone https://github.com/HKUDS/LightRAG.git
+cd LightRAG
+```
+```powershell
+# Windows PowerShell
+git clone https://github.com/HKUDS/LightRAG.git
+cd LightRAG
+```
+
+### Configure your environment:
+
+```bash
+# Linux/MacOS
+cp env.example .env
+# Edit .env with your preferred configuration
+```
+```powershell
+# Windows PowerShell
+Copy-Item env.example .env
+# Edit .env with your preferred configuration
+```
+
+LightRAG can be configured using environment variables in the `.env` file:
+
+**Server Configuration**
+
+- `HOST`: Server host (default: 0.0.0.0)
+- `PORT`: Server port (default: 9621)
+
+**LLM Configuration**
+
+- `LLM_BINDING`: LLM backend to use (lollms/ollama/openai)
+- `LLM_BINDING_HOST`: LLM server host URL
+- `LLM_MODEL`: Model name to use
+
+**Embedding Configuration**
+
+- `EMBEDDING_BINDING`: Embedding backend (lollms/ollama/openai)
+- `EMBEDDING_BINDING_HOST`: Embedding server host URL
+- `EMBEDDING_MODEL`: Embedding model name
+- `EMBEDDING_ASYMMETRIC`: Explicitly enable query/document asymmetric embeddings
+- `EMBEDDING_DOCUMENT_PREFIX`: Document prefix for prefix-based asymmetric embeddings (or `NO_PREFIX`)
+- `EMBEDDING_QUERY_PREFIX`: Query prefix for prefix-based asymmetric embeddings (or `NO_PREFIX`)
+
+See [Asymmetric Embedding Configuration](./AsymmetricEmbedding.md) for prefix
+validation rules and provider-specific behavior.
+
+**RAG Configuration**
+
+- `MAX_ASYNC_LLM`: Maximum async operations (deprecated alias: `MAX_ASYNC`)
+- `MAX_TOKENS`: Maximum token size
+- `EMBEDDING_DIM`: Embedding dimensions
+
+## 🐳 Docker Deployment
+
+Docker instructions work the same on all platforms with Docker Desktop installed.
+
+### Build Optimization
+
+The Dockerfile uses BuildKit cache mounts to significantly improve build performance:
+
+- **Automatic cache management**: BuildKit is automatically enabled via `# syntax=docker/dockerfile:1` directive
+- **Faster rebuilds**: Only downloads changed dependencies when `uv.lock` or `bun.lock` files are modified
+- **Efficient package caching**: UV and Bun package downloads are cached across builds
+- **No manual configuration needed**: Works out of the box in Docker Compose and GitHub Actions
+
+### Start LightRAG  server:
+
+```bash
+docker compose up -d
+```
+
+If you used the interactive setup, start the generated stack with:
+
+```bash
+docker compose -f docker-compose.final.yml up -d
+```
+
+The interactive setup keeps `.env` host-usable. Container-only hostnames such as `postgres` or `host.docker.internal`, along with staged SSL paths under `/app/data/certs/`, are injected into the generated `docker-compose.final.yml` for the `lightrag` service instead of being persisted back into `.env`.
+On reruns, unchanged wizard-managed service blocks in `docker-compose.final.yml` are preserved by
+default. To repair or fully regenerate those managed blocks from the bundled templates, rerun the
+matching setup target with `make env-base-rewrite` or `make env-storage-rewrite`.
+
+If the generated stack includes local Milvus, compose resolves `MINIO_ACCESS_KEY_ID` and
+`MINIO_SECRET_ACCESS_KEY` at startup from the repo `.env` or exported shell environment. The
+generated compose file does not snapshot those values, and `docker compose` exits immediately if
+either variable is missing.
+
+Before exposing the generated stack beyond localhost, run:
+
+```bash
+make env-security-check
+```
+
+That command audits the current `.env` for missing authentication, unsafe whitelist settings, weak
+JWT secrets, and other setup-level security risks without rewriting any files.
+
+LightRAG Server uses the following paths for data storage:
+
+```
+data/
+├── rag_storage/    # RAG data persistence
+└── inputs/         # Input documents
+```
+
+### Container security (non-root)
+
+The official images run the server as a non-root user (`lightrag`, uid/gid `1000`) to satisfy CIS Docker Benchmark 4.1. The behavior is designed so existing deployments keep working on upgrade:
+
+- The container starts as root, takes ownership of the writable data directories, then drops to uid 1000 via `gosu`. This means **existing root-owned bind-mount / PVC data is adopted automatically** — no manual `chown` is needed when upgrading from an older image.
+- Ownership is fixed for `/app/data` **and** any custom locations you set via `WORKING_DIR`, `INPUT_DIR`, `PROMPT_DIR`, or `TIKTOKEN_CACHE_DIR`, so pointing data outside `/app/data` still works.
+- If you instead start the container with an explicit non-root user (Compose `user: "1000:1000"` or Kubernetes `runAsUser: 1000`), the startup `chown` is skipped — make sure the mounted directories are already owned by that uid.
+- `.env` is **not** chowned, so the host keeps ownership and you can edit it freely. It only needs to be *readable* by uid 1000, which the default `0644` permission satisfies. A `.env` mounted read-only as `0600`/`0400` owned by a different uid will fail to load (clear `PermissionError` at startup); make it readable by uid 1000, or supply configuration via environment variables instead (`env_file:` / `environment:`, or k8s `env` / `envFrom`).
+
+Passing server flags still works as before, e.g. `docker run ghcr.io/hkuds/lightrag:latest --port 9622`.
+
+> Note: the image starts as root and drops privileges at runtime (the pattern used by the official PostgreSQL/Redis images), so the *running process* is non-root while the image's configured `USER` is still root. Scanners that judge CIS 4.1 purely from the `USER` directive may still flag the image even though no server process runs as root.
+
+### Optional: local vLLM embedding and reranker
+
+To run embedding and/or reranking locally with vLLM, run `make env-base` and answer `yes` when prompted to run the embedding model and rerank service locally via Docker.
+That configures the embedding service to use `BAAI/bge-m3` on port 8001 with a local vLLM server, and can also add a `vllm-rerank` service on port 8000.
+
+Alternatively, rerun `make env-base` later and enable only the rerank Docker prompt to add the `vllm-rerank` service automatically.
+vLLM provides a `v1/rerank` endpoint that works with the `cohere` binding.
+
+Example `docker-compose.override.yml` for GPU hosts (embedding + reranker):
+
+```yaml
+services:
+  vllm-embed:
+    image: vllm/vllm-openai:latest
+    runtime: nvidia
+    command: >
+      --model BAAI/bge-m3
+      --port 8001
+      --dtype float16
+    ports:
+      - "8001:8001"
+    volumes:
+      - ./data/hf-cache:/root/.cache/huggingface
+    ipc: host
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+
+  vllm-rerank:
+    image: vllm/vllm-openai:latest
+    runtime: nvidia
+    command: >
+      --model BAAI/bge-reranker-v2-m3
+      --port 8000
+      --dtype float16
+    ports:
+      - "8000:8000"
+    volumes:
+      - ./data/hf-cache:/root/.cache/huggingface
+    ipc: host
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+```
+
+For CPU-only hosts, use the official CPU image instead:
+
+```yaml
+services:
+  vllm-embed:
+    image: vllm/vllm-openai-cpu:latest
+    command: >
+      --model BAAI/bge-m3
+      --port 8001
+      --dtype float32
+    ports:
+      - "8001:8001"
+    volumes:
+      - ./data/hf-cache:/root/.cache/huggingface
+
+  vllm-rerank:
+    image: vllm/vllm-openai-cpu:latest
+    command: >
+      --model BAAI/bge-reranker-v2-m3
+      --port 8000
+      --dtype float32
+    ports:
+      - "8000:8000"
+    volumes:
+      - ./data/hf-cache:/root/.cache/huggingface
+```
+
+Add the embedding and rerank config to `.env`:
+
+```bash
+EMBEDDING_BINDING=openai
+EMBEDDING_MODEL=BAAI/bge-m3
+EMBEDDING_DIM=1024
+EMBEDDING_BINDING_HOST=http://localhost:8001/v1
+EMBEDDING_BINDING_API_KEY=local-key
+VLLM_EMBED_DEVICE=cpu
+
+RERANK_BINDING=cohere
+RERANK_MODEL=BAAI/bge-reranker-v2-m3
+RERANK_BINDING_HOST=http://localhost:8000/rerank
+RERANK_BINDING_API_KEY=local-key
+VLLM_RERANK_DEVICE=cpu
+```
+
+If LightRAG runs in Docker while vLLM runs on the host, the generated compose file rewrites those endpoints to:
+
+```bash
+EMBEDDING_BINDING_HOST=http://host.docker.internal:8001/v1
+RERANK_BINDING_HOST=http://host.docker.internal:8000/rerank
+```
+
+For GPU, set:
+
+```bash
+VLLM_EMBED_DEVICE=cuda
+VLLM_RERANK_DEVICE=cuda
+```
+
+Ensure the NVIDIA Container Toolkit is installed and the host has CUDA drivers available.
+The setup wizard uses the CPU image by default for `cpu` device and the GPU image for `cuda` device.
+When rerunning `make env-base`, an existing `VLLM_EMBED_DEVICE` / `VLLM_RERANK_DEVICE` value is
+preserved instead of being overwritten by a fresh GPU auto-detection result.
+Those templates already pin the matching vLLM `--dtype` (`float32` on CPU, `float16` on CUDA), so no separate `VLLM_*_DTYPE` environment variables are needed.
+
+### SSL certificates
+
+The setup wizard stages TLS certificate files under `./data/certs/` before generating the compose file.
+This keeps generated host mounts under the same `./data` root used by the default Docker deployment.
+
+### PostgreSQL image
+
+The interactive setup defaults PostgreSQL to `gzdaniel/postgres-for-rag:pg18-age-pgvector`. This image bundles both Apache AGE and pgvector so the generated stack works with `PGGraphStorage` and `PGVectorStorage` without extra extension setup.
+
+The image no longer ships fixed credentials; on first start it creates the user, password, and database from the `POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB` environment variables. The setup wizard prompts for these values (defaulting to `rag` / `rag` / `lightrag`) and injects them into the generated `docker-compose.final.yml`, so you can choose any user, password, and database name.
+
+**Important Note**: If PGGraphStorage is not required for vector storage, you may replace the upper docker image with the latest official pgvector image `pgvector/pgvector:pg18`. Please note that data file formats are incompatible across different PostgreSQL major versions; once this Docker image is deployed, it cannot be rolled back to a previous version.
+
+#### Build the PostgreSQL image
+
+The default PostgreSQL image can be rebuilt from the repository root with `Dockerfile.postgres`. The Dockerfile starts from `pgvector/pgvector:pg18-trixie`, builds Apache AGE for PostgreSQL 18, and installs an init script that creates both the `vector` and `age` extensions for new databases.
+
+For local development or single-host deployment:
+
+```bash
+docker build -f Dockerfile.postgres \
+  -t gzdaniel/postgres-for-rag:pg18-age-pgvector \
+  .
+```
+
+To publish the image to your own registry, use a private tag:
+
+```bash
+docker build -f Dockerfile.postgres \
+  -t registry.example.com/postgres-for-rag:pg18-age-pgvector \
+  .
+docker push registry.example.com/postgres-for-rag:pg18-age-pgvector
+```
+
+For a multi-architecture image:
+
+```bash
+docker buildx build \
+  --platform linux/amd64,linux/arm64 \
+  -f Dockerfile.postgres \
+  -t registry.example.com/postgres-for-rag:pg18-age-pgvector \
+  --push \
+  .
+```
+
+After building a custom tag, update the `postgres.image` value in the generated `docker-compose.final.yml` to point at that tag. On later setup wizard reruns, existing wizard-managed service images are preserved.
+
+### Updates
+
+To update the Docker container:
+```bash
+docker compose pull
+docker compose down
+docker compose up
+```
+
+### Offline deployment
+
+Software packages requiring `transformers`, `torch`, or `cuda` are not preinstalled in the docker images. Consequently, document extraction tools such as Docling, as well as local LLM models like Hugging Face and LMDeploy, cannot be used in an offline environment. These high-compute-resource-demanding services should not be integrated into LightRAG. Docling will be decoupled and deployed as a standalone service.
+
+## 📦 Build Docker Images
+
+### For local development and testing
+
+```bash
+# Build and run with Docker Compose (BuildKit automatically enabled)
+docker compose up --build
+
+# Or explicitly enable BuildKit if needed
+DOCKER_BUILDKIT=1 docker compose up --build
+```
+
+**Note**: BuildKit is automatically enabled by the `# syntax=docker/dockerfile:1` directive in the Dockerfile, ensuring optimal caching performance.
+
+### For production release
+
+ **multi-architecture build and push**:
+
+```bash
+# Use the provided build script
+./docker-build-push.sh
+```
+
+**The build script will**:
+
+- Check Docker registry login status
+- Create/use buildx builder automatically
+- Build for both AMD64 and ARM64 architectures
+- Push to GitHub Container Registry (ghcr.io)
+- Verify the multi-architecture manifest
+
+**Prerequisites**:
+
+Before building multi-architecture images, ensure you have:
+
+- Docker 20.10+ with Buildx support
+- Sufficient disk space (20GB+ recommended for offline image)
+- Registry access credentials (if pushing images)
+
+### Verify official GHCR images with Cosign
+
+Official LightRAG images published to GitHub Container Registry by GitHub Actions are signed with Sigstore Cosign using GitHub OIDC keyless signing.
+
+Install `cosign`, then verify the image tag you want to run:
+
+```bash
+cosign verify ghcr.io/HKUDS/LightRAG:<tag> \
+  --certificate-identity-regexp '^https://github.com/HKUDS/LightRAG/.github/workflows/(docker-publish|docker-build-manual|docker-build-lite)\.yml@refs/.+$' \
+  --certificate-oidc-issuer https://token.actions.githubusercontent.com
+```
+
+Replace `<tag>` with the version tag you want to validate, for example a release tag, `latest`, `<tag>-lite`, or `lite`.

docs/FrontendBuildGuide.md

@@ -0,0 +1,216 @@
+# Frontend Build Guide
+
+## Overview
+
+The LightRAG project includes a React-based WebUI frontend. This guide explains how frontend building works in different scenarios.
+
+## Key Principle
+
+- **Git Repository**: Frontend build results are **NOT** included (kept clean)
+- **PyPI Package**: Frontend build results **ARE** included (ready to use)
+- **Build Tool**: **Bun** is recommended, but **Node.js/npm** is fully supported as a fallback
+
+## Installation Scenarios
+
+### 1. End Users (From PyPI) ✨
+
+**Command:**
+```bash
+pip install lightrag-hku[api]
+```
+
+**What happens:**
+- Frontend is already built and included in the package
+- No additional steps needed
+- Web interface works immediately
+
+---
+
+### 2. Development Mode (Recommended for Contributors) 🔧
+
+**Command:**
+```bash
+# Clone the repository
+git clone https://github.com/HKUDS/LightRAG.git
+cd LightRAG
+
+# Install in editable mode (no frontend build required yet)
+pip install -e ".[api]"
+
+# Build frontend when needed (can be done anytime)
+cd lightrag_webui
+bun install --frozen-lockfile
+bun run build
+cd ..
+```
+
+**Advantages:**
+- Install first, build later (flexible workflow)
+- Changes take effect immediately (symlink mode)
+- Frontend can be rebuilt anytime without reinstalling
+
+**How it works:**
+- Creates symlinks to source directory
+- Frontend build output goes to `lightrag/api/webui/`
+- Changes are immediately visible in installed package
+
+---
+
+### 3. Normal Installation (Testing Package Build) 📦
+
+**Command:**
+```bash
+# Clone the repository
+git clone https://github.com/HKUDS/LightRAG.git
+cd LightRAG
+
+# ⚠️ MUST build frontend FIRST
+cd lightrag_webui
+bun install --frozen-lockfile
+bun run build
+cd ..
+
+# Now install
+pip install ".[api]"
+```
+
+**What happens:**
+- Frontend files are **copied** to site-packages
+- Post-build modifications won't affect installed package
+- Requires rebuild + reinstall to update
+
+**When to use:**
+- Testing complete installation process
+- Verifying package configuration
+- Simulating PyPI user experience
+
+---
+
+### 4. Creating Distribution Package 🚀
+
+**Command:**
+```bash
+# Build frontend first
+cd lightrag_webui
+bun install --frozen-lockfile --production
+bun run build
+cd ..
+
+# Create distribution packages
+python -m build
+
+# Output: dist/lightrag_hku-*.whl and dist/lightrag_hku-*.tar.gz
+```
+
+**What happens:**
+- `setup.py` checks if frontend is built
+- If missing, installation fails with helpful error message
+- Generated package includes all frontend files
+
+---
+
+## GitHub Actions (Automated Release)
+
+When creating a release on GitHub:
+
+1. **Automatically builds frontend** using Bun
+2. **Verifies** build completed successfully
+3. **Creates Python package** with frontend included
+4. **Publishes to PyPI** using existing trusted publisher setup
+
+**No manual intervention required!**
+
+---
+
+## Quick Reference
+
+| Scenario | Command | Frontend Required | Can Build After |
+|----------|---------|-------------------|-----------------|
+| From PyPI | `pip install lightrag-hku[api]` | Included | No (already installed) |
+| Development | `pip install -e ".[api]"` | No | ✅ Yes (anytime) |
+| Normal Install | `pip install ".[api]"` | ✅ Yes (before) | No (must reinstall) |
+| Create Package | `python -m build` | ✅ Yes (before) | N/A |
+
+---
+
+## Bun Installation
+
+If you don't have Bun installed:
+
+```bash
+# macOS/Linux
+curl -fsSL https://bun.sh/install | bash
+
+# Windows
+powershell -c "irm bun.sh/install.ps1 | iex"
+```
+
+Official documentation: https://bun.sh
+
+---
+
+## File Structure
+
+```
+LightRAG/
+├── lightrag_webui/          # Frontend source code
+│   ├── src/                 # React components
+│   ├── package.json         # Dependencies
+│   └── vite.config.ts       # Build configuration
+│       └── outDir: ../lightrag/api/webui  # Build output
+│
+├── lightrag/
+│   └── api/
+│       └── webui/           # Frontend build output (gitignored)
+│           ├── index.html   # Built files (after running bun run build)
+│           └── assets/      # Built assets
+│
+├── setup.py                 # Build checks
+├── pyproject.toml           # Package configuration
+└── .gitignore               # Excludes lightrag/api/webui/* (except .gitkeep)
+```
+
+---
+
+## Troubleshooting
+
+### Q: I installed in development mode but the web interface doesn't work
+
+**A:** Build the frontend:
+```bash
+cd lightrag_webui && bun run build
+```
+
+### Q: I built the frontend but it's not in my installed package
+
+**A:** You probably used `pip install .` after building. Either:
+- Use `pip install -e ".[api]"` for development
+- Or reinstall: `pip uninstall lightrag-hku && pip install ".[api]"`
+
+### Q: Where are the built frontend files?
+
+**A:** In `lightrag/api/webui/` after running `bun run build`
+
+### Q: Can I use npm or yarn instead of Bun?
+
+**A:** Yes. The build scripts (`dev`, `build`, `preview`, `lint`) are runtime-agnostic and work with both Bun and Node.js/npm:
+```bash
+npm install
+npm run build
+```
+Bun is recommended for speed, but npm is fully supported. Tests (`bun test`) still require Bun.
+
+### Q: Build fails with `Cannot find package '@/lib'`
+
+**A:** This was caused by `vite.config.ts` using a TypeScript path alias (`@/`) that only Bun could resolve at config load time. Update to the latest version where this is fixed with a relative import.
+
+---
+
+## Summary
+
+✅ **PyPI users**: No action needed, frontend included
+✅ **Developers**: Use `pip install -e ".[api]"`, build frontend when needed
+✅ **CI/CD**: Automatic build in GitHub Actions
+✅ **Git**: Frontend build output never committed
+
+For questions or issues, please open a GitHub issue.

docs/InteractiveSetup.md

@@ -0,0 +1,303 @@
+# Interactive Setup Guide
+
+Use the interactive setup wizard when you want LightRAG to guide you through the configuration instead of editing `.env` by hand.
+
+The wizard is exposed through `make` targets:
+
+- `make env-base`
+- `make env-storage`
+- `make env-server`
+- `make env-validate`
+- `make env-security-check`
+- `make env-backup`
+- `make env-base-rewrite`
+- `make env-storage-rewrite`
+
+You do not need to call the underlying shell script directly.
+
+## What This Wizard Is For
+
+The setup wizard helps you configure LightRAG in three parts:
+
+- `env-base` sets up the LLM, embedding model, and optional reranker.
+- `env-storage` adds or changes storage backends such as PostgreSQL, Neo4j, Redis, Milvus, Qdrant, MongoDB, or Memgraph.
+- `env-server` sets server host and port, WebUI labels, authentication, API keys, and SSL.
+
+You can rerun each step later. The wizard loads your existing `.env` and shows current values as defaults, so you only need to change what is different.
+
+## Before You Start
+
+- Run commands from the repository root.
+- The `make env-*` targets automatically choose a compatible Bash 4+ interpreter.
+- Use the documented `make env-*` targets rather than invoking the setup script yourself.
+- `make env-base` is the normal starting point because it creates the initial `.env`.
+- `make env-storage` and `make env-server` require an existing `.env`.
+- If you choose any wizard-managed Docker service, the wizard also prepares LightRAG for the Docker startup path.
+
+## Choose Your Setup Path
+
+Use this quick guide to decide what to run:
+
+- I want the fastest first run with remote model providers: `make env-base`
+- I want embedding or reranking to run locally in Docker: `make env-base`
+- I already configured models and now want databases: `make env-storage`
+- I already configured models and now want auth, API keys, or SSL: `make env-server`
+- I want to check whether my current setup is valid: `make env-validate`
+- I want to audit my current setup before exposing it: `make env-security-check`
+- I want a standalone backup without changing configuration: `make env-backup`
+- I need to repair the generated compose services from the bundled templates: `make env-base-rewrite` or `make env-storage-rewrite`
+
+## Scenario 1: First-Time Local Setup
+
+Use this when you want LightRAG running with the least amount of setup and you already have remote model endpoints or API keys.
+
+**Command**
+
+```bash
+make env-base
+```
+
+**What the wizard asks**
+
+- LLM provider, model, endpoint, and API key
+- Whether the embedding model should run locally via Docker
+- If embedding stays remote: embedding provider, model, dimension, endpoint, and API key
+- Whether reranking should be enabled
+- If reranking is enabled: whether the rerank service should run locally via Docker
+- If reranking stays remote: rerank provider, model, endpoint, and API key
+
+**What gets written**
+
+- `.env`
+- `docker-compose.final.yml` only if you enabled wizard-managed Docker services
+
+**What to do next**
+
+- If you did not enable wizard-managed Docker services:
+
+```bash
+lightrag-server
+```
+
+- If you enabled wizard-managed Docker services:
+
+```bash
+docker compose -f docker-compose.final.yml up -d
+```
+
+## Scenario 2: Local Setup With Docker-Hosted Embedding or Rerank
+
+Use this when you want LightRAG to run local inference services for embedding and/or reranking through Docker.
+
+**Command**
+
+```bash
+make env-base
+```
+
+**Recommended answers**
+
+- Answer `yes` to `Run embedding model locally via Docker (vLLM)?` if you want local embeddings
+- Answer `yes` to `Enable reranking?` and then `yes` to `Run rerank service locally via Docker?` if you want local reranking
+
+**What the wizard asks after you enable local services**
+
+- Embedding model name for local vLLM
+- Rerank model name for local vLLM
+- Remote LLM details if your main LLM is still external
+
+**What gets written**
+
+- `.env`
+- `docker-compose.final.yml` with the selected local services
+
+**What to do next**
+
+```bash
+docker compose -f docker-compose.final.yml up -d
+```
+
+This starts the generated Docker-based LightRAG stack together with the selected local services.
+
+## Scenario 3: Add Storage After The Base Setup
+
+Use this when you already have `.env` from `make env-base` and now want to switch from default local-file storage to database-backed storage.
+
+**Command**
+
+```bash
+make env-storage
+```
+
+**Prerequisite**
+
+- `.env` must already exist
+
+**What the wizard asks**
+
+- KV storage backend
+- Vector storage backend
+- Graph storage backend
+- Doc-status storage backend
+- For each required database, whether it should run locally via Docker
+- For each required database, the needed connection details such as host, URI, port, user, password, database name, or device type
+
+**Important rule**
+
+- `MongoVectorDBStorage` requires Atlas Search / Vector Search support.
+- If you choose the wizard-managed Docker MongoDB service, the wizard now provisions MongoDB Atlas Local, so `MongoVectorDBStorage` can run against the local Docker deployment. The generated host-side `MONGO_URI` uses `?directConnection=true`.
+- If you do not use the wizard-managed Docker MongoDB service, provide an external Atlas-capable MongoDB endpoint for `MONGO_URI`, such as a `mongodb+srv://` Atlas cluster URI or an Atlas Local `mongodb://...?...directConnection=true` URI.
+- For external `mongodb://...?...directConnection=true` URIs, the wizard can only validate the URI format. It cannot determine statically whether the target deployment actually provides Atlas Search / Vector Search support.
+
+**What gets written**
+
+- `.env`
+- `docker-compose.final.yml` if you selected wizard-managed storage services
+
+**What to do next**
+
+- If you selected Docker-managed storage services:
+
+```bash
+docker compose -f docker-compose.final.yml up -d
+```
+
+- If you pointed LightRAG at external databases, make sure those services are reachable before starting LightRAG.
+
+## Scenario 4: Harden A Deployment With Auth And SSL
+
+Use this when you already have `.env` and need to prepare the server for shared or external use.
+
+**Commands**
+
+```bash
+make env-server
+make env-security-check
+```
+
+**Prerequisite**
+
+- `.env` must already exist
+
+**What `env-server` asks**
+
+- Server host and port
+- WebUI title and description
+- Summary language
+- Whether to configure authentication and API key settings
+- Auth accounts, JWT secret, token lifetime, API key, and whitelist paths
+- Whether to enable SSL/TLS
+- SSL certificate file path and SSL key file path
+
+**What gets written**
+
+- `.env`
+- `docker-compose.final.yml` may be updated if your current setup already uses wizard-managed Docker services
+
+**What to do next**
+
+- Run `make env-security-check`
+- If the stack uses Docker, recreate the LightRAG service with your compose file
+- If the stack runs on the host, restart `lightrag-server`
+
+For broader deployment guidance, see [DockerDeployment.md](./DockerDeployment.md).
+
+## Validate, Audit, And Backup
+
+These commands do not walk you through a full setup flow, but they are part of normal operations.
+
+### Validate The Current Configuration
+
+```bash
+make env-validate
+```
+
+Use this when you want to confirm that the current `.env` is internally consistent. It reports problems such as missing required values, malformed auth settings, invalid URIs, invalid ports, or missing SSL files.
+
+### Audit Security Before Exposure
+
+```bash
+make env-security-check
+```
+
+Use this before exposing LightRAG beyond localhost. It reports risky setups such as missing authentication, weak or missing JWT secrets, unsafe whitelist settings, or unresolved sensitive placeholders.
+
+### Create A Standalone Backup
+
+```bash
+make env-backup
+```
+
+Use this when you want a manual backup without running any setup flow.
+
+## Outputs And What They Mean
+
+### `.env`
+
+The wizard writes `.env` in the repository root. This file becomes the current runtime configuration produced by the latest wizard run.
+
+In practice, this means:
+
+- rerunning the wizard updates `.env`
+- existing values are reused as defaults on later runs
+- you should treat `.env` as the active configuration for the workflow you most recently configured
+- before `env-base`, `env-storage`, or `env-server` writes `.env`, the wizard automatically creates a timestamped backup of the existing file when one is present
+
+### `docker-compose.final.yml`
+
+The wizard creates or updates `docker-compose.final.yml` only when you choose wizard-managed Docker services or when an existing wizard-generated compose setup needs to stay aligned with new server settings.
+
+When one of the setup flows is about to replace or remove an existing generated compose file, it automatically creates a timestamped backup first.
+
+For MongoDB-backed storage, the wizard-managed Docker path uses MongoDB Atlas Local rather than MongoDB Community Edition so local Atlas Search / Vector Search workflows are available.
+
+Use this file when starting the generated Docker stack:
+
+```bash
+docker compose -f docker-compose.final.yml up -d
+```
+
+The base `docker-compose.yml` remains the general project compose file. The generated `docker-compose.final.yml` is the wizard-managed output.
+
+## Troubleshooting And Advanced Notes
+
+- If `make env-storage` or `make env-server` says `.env` is missing, run `make env-base` first.
+- You do not need to run `make env-backup` before rerunning `env-base`, `env-storage`, or `env-server`; those flows already back up the existing `.env`, and they also back up the generated compose file before changing it.
+- If you need to fully rebuild wizard-managed compose services from the current bundled templates, use `make env-base-rewrite` or `make env-storage-rewrite`.
+- If you switch between host-oriented and Docker-oriented workflows, rerun the relevant setup step instead of trying to manually merge old settings.
+- If the generated stack includes local Milvus, make sure `MINIO_ACCESS_KEY_ID` and `MINIO_SECRET_ACCESS_KEY` are available before running `docker compose -f docker-compose.final.yml up -d`.
+- For Docker deployment details beyond the interactive wizard, see [DockerDeployment.md](./DockerDeployment.md).
+
+## Typical Command Sequences
+
+### Remote models, local server
+
+```bash
+make env-base
+lightrag-server
+```
+
+### Remote LLM, local embedding and rerank in Docker
+
+```bash
+make env-base
+docker compose -f docker-compose.final.yml up -d
+```
+
+### Add storage after the base setup
+
+```bash
+make env-base
+make env-storage
+docker compose -f docker-compose.final.yml up -d
+```
+
+### Add security and SSL before exposure
+
+```bash
+make env-base
+make env-storage
+make env-server
+make env-security-check
+docker compose -f docker-compose.final.yml up -d
+```

docs/LightRAGSidecarFormat-zh.md

@@ -0,0 +1,402 @@
+# LightRAG Sidecar 文件格式说明
+
+本文介绍内解析引擎输出的**LightRAG Sidecar**文件格式。LightRAG 在使用native/mineru/docling这些支持多模态内容解析引擎提取文件内容的时候，会把"正文 + 多模态对象 + 解析元数据"拆开写到一个 `*.parsed/` 目录中，目录内的每个 JSON / JSONL 文件统称为 **sidecar** 文件。Sidecar 是后续流水线（多模态分析 → 多模态 chunk 构造 → 实体抽取 → 文档删除时的缓存清理）唯一可靠的依据。Sidecar的文件格式是LightRAG内置的通用文件交换格式，新的多模态内容提取引擎都需要遵循这个格式。公开**LightRAG Sidecar**文件格式的目的是给社区开发者编写字节的内容解析引擎提供方便。
+
+## 一、概述
+
+| 关注点 | 文件 | 存放内容 | 说明 |
+|---|---|---|---|
+| 主文件 | `<doc>.blocks.jsonl` | 存放 Block 正文 | 所有 Block 的 content字段内容拼接后形成完整的原文 |
+| 图形对象 | `<doc>.drawings.json` | 文件中抽取出来的图形对象 | 送VLM进行分析后回填分析结果 |
+| 表格对象 | `<doc>.tables.json` | 文件中抽取出来的表格对象 | 送LLM进行分析后回填分析结果 |
+| 公式对象 | `<doc>.equations.json` | 文件中抽取出来的公司对象 | 送LLM进行分析后回填分析结果 |
+| 原始图像资源 | `<doc>.blocks.assets/` | 文件中抽取出来的图片原始文件 | 送VLM进行图片分析 |
+
+Sidecar 的设计意图：
+
+- 解析阶段 内容提取引擎(native/mineru/docling) **只**负责生成 `blockid / heading / content / surrounding` 等"客观"字段；
+- 多模态分析阶段 (`analyze_multimodal`) 由 LightRAG 写入分析结果 `llm_analyze_result` 字典，可能是首次追加，也可能覆盖已有结果；解析器不应预先填充该字段
+
+## 二、目录布局
+
+```
+inputs/space1/__parsed__/<规范文件名>.parsed/
+├── <规范文件名>.blocks.jsonl        正文块序列 + 文档级 meta（首行）
+├── <规范文件名>.drawings.json       图形 sidecar（dict 容器，键 = 图形 id）
+├── <规范文件名>.tables.json         表格 sidecar
+├── <规范文件名>.equations.json      公式 sidecar
+└── <规范文件名>.blocks.assets/      原始资源目录（存放drawings.json中的图片文件放这里）
+    ├── image1.wmf
+    ├── image2.wmf
+    ├── image3.wmf
+    ├── image4.png
+    ├── image5.png
+    ├── image6.png
+    └── image7.emf
+```
+
+## 三、blocks.jsonl
+
+`blocks.jsonl` 是按行序列化的 JSON，**第一行 `type="meta"`**，其余每行是一个内容块 `type="content"`。
+
+### 3.1 meta 行实例
+
+```json
+{
+  "type": "meta",
+  "format": "lightrag",
+  "version": "1.0",
+  "document_name": "m012-manual.docx",
+  "document_format": "docx",
+  "document_hash": "sha256:4840...3f9543d9db0822d2d59",
+  "table_file": true,
+  "equation_file": true,
+  "drawing_file": true,
+  "asset_dir": true,
+  "blocks": 39,
+  "doc_id": "doc-f1bee60173d067d88595c00e7d9b0ce5",
+  "parse_engine": "native",
+  "parse_time": "2026-05-13T18:42:25.943490+00:00",
+  "doc_title": "m012-manual"
+}
+```
+
+| 字段 | 类型 | 说明 |
+|---|---|---|
+| `type` | `"meta"` | 行类型，固定值，校验位 |
+| `format` | `"lightrag"` | sidecar 大版本族标识 |
+| `version` | `str` | sidecar schema 版本 |
+| `document_name` | `str` | 规范文件名（含后缀，不含处理指示） |
+| `document_format` | `str` | 文件格式（目前以文件后缀表示） |
+| `document_hash` | `"sha256:<hex>"` | sidecar 正文指纹，定义为 `SHA-256(merged_text)`，其中 `merged_text` 是所有非空 content 行的 `content` 字段按 `"\n\n"` 拼接后的字符串。供外部消费者快速判断两份 `.parsed/` 是否同源（不必逐行比对 body），并作为 sidecar 文件的自描述内容校验位。注意：LightRAG 入库流水线本身不读此字段，跨文档去重由 `doc_status.content_hash` 单独承担 |
+| `table_file` / `equation_file` / `drawing_file` | `bool` | 是否存在对应 sidecar 文件（为真时对应文件必然存在） |
+| `asset_dir` | `bool` | 是否存在`blocks.assets`资源目录 |
+| `split_option` | `object` | 可选。解析引擎自我记录的元数据（如 `engine_version` 及引擎特定附加信息）；引擎无任何记录时（native/markdown 常见情形）整个字段省略。分块不在解析阶段进行（由下游 chunker 负责），故此字段从不含分块参数 |
+| `blocks` | `int` | content 行数（不含 meta） |
+| `doc_id` | `"doc-<md5>"` | 文档全局 id。sidecar item id（`im-/tb-/eq-`）使用 `doc_id` 去掉 `doc-` 前缀后的哈希部分，以缩短嵌入正文中的占位标签 |
+| `parse_engine` | `str` | 解析引擎`native/mineru/docling/legacy` |
+| `parse_time` | `str` | 解析完成时间; 格式：ISO-8601 UTC |
+| `doc_title` | `str` | 文档标题（通常为首个 H1）；可选 |
+| `doc_summary` | `str` | 文档摘要；可选 |
+| `doc_attributes` | `object` | 文章扩展属性对象；可选 |
+| `bbox_attributes` | `object` | bbox possition全局属性；详见[§八](八、positions) |
+
+> LightRAG要求同一个workspace（知识库）内的文件名（document_name）必须唯一。
+
+### 3.2 content 行
+
+每个 content 行是一个原始文档"块"的最小可寻址单位，至少包含：
+
+```json
+{
+  "type": "content",
+  "blockid": "b5b5264333943cfd623710da1c36fc93",
+  "format": "plain_text",
+  "content": "# 1 产品用途和功能\nMI012模块用于支撑供氧抗荷调节器的供氧抗荷控制功能...",
+  "heading": "1 产品用途和功能",
+  "parent_headings": [],
+  "level": 1,
+  "session_type": "body",
+  "table_slice": "none",
+  "positions": [
+    {
+      "type": "paraid",
+      "range": ["5EA4577A", "6555DDCB"]
+    }
+  ]
+}
+```
+
+| 字段 | 含义 |
+|---|---|
+| `type` | `"content"` |
+| `blockid` | 全局唯一的Block ID |
+| `format` | 内容形态，目前固定为 `"plain_text"` |
+| `content` | 文本内容；**公式和图片此以占位标签出现，表格以带table标签的JSON或HTLM格式出现**（见 3.3）。标题行会按 `level` 渲染 markdown `#` 前缀（后跟一个空格）：level 1 → 一个 `#`，level 2 → 两个 `#`，……，封顶 6 个（level ≥ 7 的标题仍渲染为 `######`）。若源标题文字本身已以 markdown 前缀开头（1–6 个 `#` 后跟空格），则原样保留、不再重复加前缀。注意 `heading` 字段本身保持干净（不含 `#`）。 |
+| `heading` | content所在章节的最高层级标题；heading真实存在时，应该以 markdown 渲染后的标题行出现在content的开头。**每个被识别出的标题都独立成块**：若标题后紧跟正文，正文与该标题合并进同一个块（content = markdown 渲染后的标题行 + 正文）；若标题后没有正文（例如其后紧接下一个层级的标题），该标题仍单独成块、content 仅为 markdown 渲染后的标题行。这样可保证所有 Block 的 content 字段拼接后仍能完整、无重叠地还原原文。 |
+| `parent_headings` | 字符串数组: 自顶向下的祖先标题列表，不含当前 `heading` |
+| `level` | 整数: `heading` 在文档大纲中的层级（`1` = H1 / 一级标题，0表示无标题） |
+| `session_type` | Block所处区域：`body` `preface` `TOC` `references` `appendix` |
+| `table_slice` | 可选保留字段；表示Block是否仅包括表格片段。目前分析引擎不会拆分长表格。因此本字段固定为 `"none"`（表示表格不会被分片） |
+| `table_header` | 可选保留字段；在当前块位表格片段的时候，保存识别出来的表格头。目前不存在 |
+| `positions` | `position` 对象数组：标识文本块的版面位置；文本块来与版面的多个位置的时候，则会出现多个`position` 对象。参见[§八](#八、position) |
+
+> - blockid计算方式：`md5(doc_id + ":" + block_index + ":" + heading + ":" + content)`。文档经过分块策略处理得到的 chunk 将保存 blockid 用于溯源 chunk 在s idecar 中的位置。
+> - 不关系文档章节结构的分块策略 `F` `R` `V` 使用的就是 content 字段拼接后的内容进行分块。因此需要保证所有 Block 的 content字段合并在一起能够构成完整的文档内容，不会缺少内容，不会出现重叠的内容。
+
+### 3.3 content 内嵌占位标签
+
+为了让 P 分块策略在不破坏多模态对象的前提下对正文做切分，`content` 文本里使用如下三种 XML 风格的占位标签：
+
+| 标签 | 含义 | 标签属性 |
+|---|---|---|
+| `<table id="tb-…" format="json">…</table>` | 表格占位，包体是表格原始 JSON / HTML | `id` 指向 `tables.json` 里对应 item；`format` ∈ `json` / `html` |
+| `<drawing id="im-…" format="png" path="…" src="…" caption="…" />` | 自闭合图形占位 | `id` 指向 `drawings.json`；`path` 相对 `*.parsed/` 目录；`src` 是原文档里的引用名 |
+| `<equation id="eq-…" format="latex" caption="…">…</equation>` | 公式占位 | 行内公式同样用 `<equation format="latex">` 但**不**带 `id`，不会进 sidecar； 仅块公式（独占一行或多行）时携带 `id` |
+
+在实体关系抽取的时候喂给大模型的文本会把 `id / path / src` 等内部属性剥掉，但为保留键属性（`format / caption`）。目的是避免抽取出文章不可见的实体，给抽取结果注入过多的噪声。
+
+### 3.4 blockid 与 chunk sidecar.refs 的对应
+
+葛总分块策略在sidecar文件存在时，会在其输出的每个 chunk 都会带上 `sidecar = {"type": "block", "id": <主来源 blockid>, "refs": [{"type": "block", "id": <blockid>}, …]}`，其中：
+
+- 未合并的 chunk → `sidecar.refs` 只有一个元素，等于该 chunk 来自的 blocks.jsonl 行的 `blockid`；
+- LevelMerge 合并后的 chunk → `refs` 顺序保留所有来源 `blockid`（去重）；
+- hard fallback split 后的子 chunk → 共享父 chunk 的 `sidecar`。
+
+这条链路是文档级追溯（chunk ↔ block ↔ 原段落 paraId）的基础。
+
+## 四、drawings.json
+
+顶层是 `{"version": "1.0", "drawings": { <id>: <item>, … }}` 形态的 dict 容器，**键 = `id` 字段**，便于按 id 查找。每个 item 形如：
+
+```json
+{
+  "id": "im-f1bee60173d067d88595c00e7d9b0ce5-0004",
+  "blockid": "2f52b70839d13a936d97955916820147",
+  "heading": "2.3 结构尺寸及重量",
+  "parent_headings": ["2 产品说明"],
+  "format": "png",
+  "path": "m012-manual.blocks.assets/image4.png",
+  "src": "",
+  "caption": "",
+  "footnotes": [],
+  "extras": {
+    "ocr_texts": "图内第一段 OCR 文本\n\n图内第二段 OCR 文本",
+    "ocr_texts_count": 2
+  },
+  "surrounding": {
+    "leading": "2.3 结构尺寸及重量\n尺寸及重量要求如下：\na) 外廓尺寸长度为：<drawing …",
+    "trailing": "\n图1　外廓尺寸示意\nb) 重量不大于0.85kg。\nc) 测试结果：实测电路噪声Vpp=1.526mV…"
+  },
+  "llm_analyze_result": {
+    "name": "产品外廓尺寸工程图纸",
+    "type": "Illustration",
+    "description": "该图纸为产品的外廓尺寸示意图，展示了一个电子设备或电源模块的三视图设计…",
+    "analyze_time": 1778697752,
+    "status": "success",
+    "message": ""
+  },
+  "llm_cache_list": [
+    "default:analysis:fcf4c4f88227ee1c1bf0ed4394039e37"
+  ]
+}
+```
+
+| 字段 | 说明 |
+|---|---|
+| `id` | `im-<doc_hash>-<NNNN>` 形式（`doc_hash` 为 `doc_id` 去掉 `doc-` 前缀后的 32 位 md5） |
+| `blockid` | 指向产生该图形的 content 行 |
+| `heading` | 所在章节标题 |
+| `parent_headings` | 字符串数组：自顶向下的祖先标题列表，不含当前 `heading`（与该图形所属 block 在 `blocks.jsonl` 中的同名字段一致） |
+| `format` | 原始扩展名（去点）：`png` / `jpeg` / `gif` / `webp` / `wmf` / `emf` / … |
+| `path` | 相对 `*.parsed/` 目录的资源路径，**永远**指向 `*.blocks.assets/` 内文件 |
+| `src` | 原文档里图形的引用别名（多数情况下为空） |
+| `caption` | 可见标题（解析器可能留空） |
+| `footnotes` | 脚注字符串列表 |
+| `surrounding` | 上下文对象：参见[§七](#七、surrounding) |
+| `self_ref` | 字符串：可选；解析引擎原始输出中的对象引用（如 Docling JSON Pointer `#/pictures/3`，或 MinerU `content_list.json#/23`），用于溯源时回查原始解析产物中的对应对象（页面位置、原始结构等）。native 等不提供此字段时不输出 |
+| `extras` | 对象：可选；引擎专属的旁路字段（如图片中包含的OCR文字等）。不属于 spec 校验范围，下游消费者不应依赖具体键。 |
+| `llm_analyze_result` | 模态分析结果对象：详见 [§九](#九、`llm_analyze_result`) （后续会注入到多模态文本块） |
+| `llm_cache_list` | 模态分析LLM缓存数组（后续会注入到多模态文本块） |
+
+`extras` 中常见的 drawing 专属键：
+
+| 键 | 说明 |
+|---|---|
+| `ocr_texts` | 字符串：可选；图形对象内部 OCR 文本，多个段落用空行（`\n\n`）拼接。仅当解析引擎显式把 OCR 文本挂在该 drawing 的 children 下时输出；caption / footnote 不进入此字段。 |
+| `ocr_texts_count` | 整数：可选；写入 `ocr_texts` 的非空 OCR 段落数量。 |
+
+**只有图形支持的 raster 格式（png / jpeg / gif / webp）才会进入 VLM 分析**；其他格式（wmf / emf / svg 等）写 `llm_analyze_result.status="skipped"`，下游不生成多模态 chunk，文档继续处理。图片大小超过环境变量`VLM_MAX_IMAGE_BYTES`规定的大小后，图片同样不会进入VLM分析。
+
+> 图片的大小、DPI等信息统一放进 `extras` 对象；不要在 item 顶层引入未声明的字段（比如 `image` / `img_path` 等）。tables / equations 也遵循同样的 `extras` 约定。`self_ref` 是 spec 顶层声明的可选字段，不属于 extras 范围。
+
+## 五、tables.json
+
+顶层是 `{"version": "1.0", "tables": { <id>: <item>, ... }}` 形态的 dict 容器，**键 = `id` 字段**，便于按 id 查找。每个 item 形如：
+
+```json
+{
+  "id": "tb-f1bee60173d067d88595c00e7d9b0ce5-0007",
+  "blockid": "3f33897b5e105d254addc655f1efbf8c",
+  "heading": "2.4.4 温度-湿度-高度（随系统进行）",
+  "parent_headings": ["2 产品说明", "2.4 环境适应性"],
+  "dimension": [16, 8],
+  "format": "json",
+  "content": "[[\"试验步骤\", \"温度(℃)\", \"高度(m)\", \"相对湿度\", \"时间(min)\", \"辅助冷却\", \"系统电源\", \"功能、性能检查\"],…",
+  "caption": "",
+  "footnotes": [],
+  "table_header": "[[\"试验步骤\", \"温度(℃)\", \"高度(m)\", \"相对湿度\", \"时间(min)\", \"辅助冷却\", \"系统电源\", \"功能、性能检查\"]]"
+  "surrounding": {
+    "leading": "2.4.4 温度-湿度-高度（随系统进行）\n产品应能承受执行任务期间的温度、湿度、高度环境综合作用…",
+    "trailing": "\n注：以上步骤重复10个循环。a成品及附件达到温度稳定或240min，以长者为准；b成品及附件达到温度稳定或120min，以长者为准。…"
+  },
+  "llm_analyze_result": {
+    "name": "文档管理元数据表",
+    "description": "这是一份文档管理信息表，用于记录技术文档的基本元数据和版本控制信息 …",
+    "analyze_time": 1778697759,
+    "status": "success",
+    "message": ""
+  },
+  "llm_cache_list": [
+    "default:analysis:b316aacd40fdca0cb56430870bb89a62"
+  ]
+}
+```
+
+tables.json 文件的 `blockid` `heading` `parent_headings` `surrounding` `llm_analyze_result` 字段与drawings.json相同。不同或新添加的字段说明如下：
+
+| 字段 | 说明 |
+|---|---|
+| `id` | `tb-<doc_hash>-<NNNN>` 形式（`doc_hash` 为 `doc_id` 去掉 `doc-` 前缀后的 32 位 md5） |
+| `dimension` | 整数数组：`[num_rows, num_cols]`，包含表头行 |
+| `format` | `"json"` (二维数组) 或 `"html"` (负载 `<table>…</table>` 片段，含起止标签) |
+| `content` | 字符串：表格正文，按 `format` 决定结构；这是后续多模态 chunk 真正使用的字符串。 |
+| `table_header` | 字符串：可选；识别出来的作为表格头的行内容 |
+| `self_ref` | 可选；解析引擎原始输出中的对象引用（如 Docling JSON Pointer `#/tables/2`，或 MinerU `content_list.json#/31`），用于溯源时回查原始解析产物 |
+
+在模态分析阶段，如果`content`字段长度超过大模型的上下文长度时，表格内容会被机械地截断后在喂给模型。
+
+## 六、equations.json
+
+顶层是 `{"version": "1.0", "equations": { <id>: <item>, ... }}` 形态的 dict 容器，**键 = `id` 字段**，便于按 id 查找。每个 item 形如：
+
+```json
+{
+  "id": "eq-f1bee60173d067d88595c00e7d9b0ce5-0001",
+  "blockid": "2f52b70839d13a936d97955916820147",
+  "heading": "2.3 结构尺寸及重量",
+  "parent_headings": ["2 产品说明"],
+  "format": "latex",
+  "content": "C=2∗\\frac{P∗T}{\\left( {V}_{H}^{2}−{V}_{L}^{2} \\right)∗η}",
+  "caption": "",
+  "footnotes": [],
+  "surrounding": {
+    "leading": "2.3 结构尺寸及重量\n尺寸及重量要求如下：\n …",
+    "trailing": "\n其中P为供电异常时维持的功率28W,T为期望储能时间，V<sub>H</sub>为电容放电前…"
+  },
+  "llm_analyze_result": {
+    "name": "电容储能时间计算公式",
+    "description": "该公式用于计算在电源异常情况下维持系统正常工作所需的电容储能值 …",
+    "analyze_time": 1778697783,
+    "status": "success",
+    "message": "",
+    "equation": "C=2\\cdot\\frac{P\\cdot T}{(V_{H}^{2}-V_{L}^{2})\\cdot\\eta}"
+  },
+  "llm_cache_list": [
+    "default:analysis:fcf4c4f88227ee1c1bf0ed4394039e37"
+  ]
+}
+```
+
+equations.json 文件的 `blockid` `heading` `parent_headings` `surrounding` `llm_analyze_result` 字段与drawings.json相同。不同或新添加的字段说明如下：
+
+| 字段 | 说明 |
+|---|---|
+| `id` | `eq-<doc_hash>-<NNNN>` 形式（`doc_hash` 为 `doc_id` 去掉 `doc-` 前缀后的 32 位 md5） |
+| `format` | 固定为 `"latex"` |
+| `content` | 字符串：是**原始** LaTeX（可能包含 Unicode 运算符、外层 `\[ \]`），不包含两头的`$`分割符；模态分析阶段直接读这里 |
+| `self_ref` | 可选；解析引擎原始输出中的对象引用（如 Docling JSON Pointer `#/texts/15`，或 MinerU `content_list.json#/45`），用于溯源时回查原始解析产物 |
+| `llm_analyze_result.equation` | 字符串：是大模型输出的**规范化**后的 LaTeX公式（外层 `$ / \[ \] / equation` 环境，Unicode 转 LaTeX，不包含联投的`$`分割符），这是后续多模态 chunk 真正使用的字符串； |
+
+在模态分析阶段，如果`content`字段长度超过大模型的上下文长度时，表格内容会被机械地截断后在喂给模型。行内公式（与正文连续的 `<equation format="latex">…</equation>`）**不会**保存到 equations.json 文件，它仅会在 blocks 文本里以无 `id` 形式留存。这样做的目的是避免给抽取结果注入过多的噪音。
+
+## 七、surrounding
+
+`surrounding.leading` 和 `surrounding.trailing` 是 sidecar item 的可分析上下文窗口，目的是提供图片、表格和公式所在段落的上下文信息，提高多模态分析的质量。**surrounding内容有LightRAG在分析阶段自动注入，不需要在文档解析引擎中主动写入sidecar文件中**。以下是surrounding内容的生成逻辑：
+
+- 取自同一 `blockid` 对应的 content 行文本，以多模态占位标签的位置为切分点；
+- 每一侧的 token 上限由环境变量 `SURROUNDING_LEADING_MAX_TOKENS` / `SURROUNDING_TRAILING_MAX_TOKENS` 控制（缺省 `2000`，可独立调整）；按 tokenizer 截断，倾向保留靠近目标的句子；
+- 文本中保留**同行其他**多模态对象的占位标签，这让模型能感知"图 1 之后还有公式 1"这种上下文；但解析器内部标识符（`id` / `path` / `src` / `refid`）已被 `strip_internal_multimodal_markup_for_extraction` 剥离 —— 与 chunk content 实体抽取前的清理一致，避免噪声进入 VLM/LLM prompt。具体清理规则：
+  - `<drawing id="im-…" path="…" src="…" caption="Fig 1" />` → `<drawing caption="Fig 1" />`；**没有 caption 的 drawing 整段移除**（标签不再携带任何对模型可见的信息）；
+  - `<table id="tb-…" format="json" caption="…">rows</table>` → `<table format="json" caption="…">rows</table>`；
+  - `<equation id="eq-…" format="latex">body</equation>` → `<equation format="latex">body</equation>`；
+  - `<cite type="table" refid="tb-…">表 1</cite>` → `<cite type="table">表 1</cite>`；`<cite type="equation" refid="eq-…">公式 2</cite>` → `<cite type="equation">公式 2</cite>`。仅删 `refid` 属性，保留 `<cite type="…">…</cite>` 包装 —— 让 VLM/LLM 能识别"这是对其他表/公式的引用"而非普通的文本，同时屏蔽 LLM 看不到的解析器内部 id；
+    - 例外：`tables.json` 类型的 surrounding 在 strip 之前先走 `remove_table_tags`，把所有 `<cite type="table">` 整段移除（分析目标表时不希望被对其他表的悬挂引用干扰）；
+- 清理发生在 token 预算截断**之前**：token 数按"LLM 实际看到的内容"统计，且截断点不会落到未清理的 `id="…"` 属性中间，避免标签结构残缺；
+- 当目标对象本身位于 block 起点 / 终点时，对应一侧为 `""` 而不是 `"n/a"`（提示词组装时再把空字符串显示为 `n/a`）；
+- `enrich_sidecars_with_surrounding` 是幂等的：每次 `analyze_multimodal` 入口都会重新计算并覆盖 `surrounding`，因此修改 `SURROUNDING_LEADING_MAX_TOKENS` / `SURROUNDING_TRAILING_MAX_TOKENS` 后无需手动清理 sidecar，重新执行多模态分析即可按新预算重写。
+
+## 八、positions
+
+`positions`是一个对象数组，用于标识`blockid`的内容来之文件中的哪一个文字，用于内容溯源的时候能够在原始文件中找到和显示对应的内容。当`blockid`的内容是由版面的多个栏目合并而成时，会出现多个`position` 对象，每个`position` 对象对应1个版面方框或栏目。为了适应不同的文档格式的内容定位方式，系统提供了以下几种`position` 对象对象类型。
+
+`position` 对象有多种类型，对象的`type`字段决定了其类型：
+
+* paraid
+
+适用于docx格式文件；按`段落id`（paraid）定位内容。`rang`字段指定起止`段落id`；`charspan`为可选字段，指定内容从段落的m个字符开始到底n个字符结束。不提供`charspan`表示`blockid`为起止段落的全部内容。示例：
+
+```
+"positions": [
+{
+    "type": "paraid",
+    "range": ["5EA4577A", "6555DDCB"]
+    "charspan": [10,999]
+}]
+```
+
+* bbox
+
+适用于与PDF格式类似的文件，通过页面矩形位置来标定内容来源的原始位置。bbox支持一下字段：
+
+```
+origin: 矩形坐标相对于页面那个位置（可选字段，默认为LEFTTOP，另一个可选值为LEFTBOTTOM）
+max: 页面布局的长和宽的最大值，坐标按此值归一化以便能准确显示位置（可选字段，为空表示坐标按图片的点阵计算）
+anchor: 页码, 页码为字符串，支持罗马数字等非数阿拉伯数字字页码
+range: 矩形坐标数组 [h1,w1,h2,w2]，例如 [174, 155, 818, 333]
+charspan: 内容从标定段落的m个字符开始到底n个字符结束（可选字段）
+```
+
+`blocks.jsonl`文件的`meta`行的`bbox_attributes`字段保存的是bbox的全局设置，避免每个`content`行的`positions`对象中重复保存相同的内容。一下是一个典型的`positions`对象示例：
+
+```
+"positions": [
+{
+    "type": "bbox",
+    "anchor": "ii"
+    "range": [174, 155, 818, 333]
+    "charspan": [10, 999]
+}]
+```
+
+* heading
+
+适用于与Markdown格式类似的文件，按标题定位内容。`anchor`是起始标题（标题重复是的处理方式查到markdown anchor规范）；`charspan`为可选字段，指定内容从段落的m个字符开始到底n个字符结束。不提供`charspan`表示`blockid`为起止段落的全部内容。
+
+```
+"positions": [
+{
+    "type": "heading",
+    "anchor": "ii"
+    "range": [174, 155, 818, 333]
+    "charspan": [10, 999]
+}]
+```
+
+* absolute
+
+适用于text格式类似的文件，按字符绝对位置定位。`charspan`指定内容从段落的m个字符开始到底n个字符结束。
+
+```
+"positions": [
+{
+    "charspan": [10, 999]
+}]
+```
+
+## 九、`llm_analyze_result`
+
+| `status` | 触发场景 | 字段说明 |
+|---|---|---|
+| `success` | 模型成功返回合法 JSON 且必需字段齐全 | 图形：`name / type / description`；表格：`name / description`；公式：`name / description / equation` |
+| `skipped` | 期跳过多模态分析：图片格式不支持、像素 < `VLM_MIN_IMAGE_PIXEL`（默认 32px）、大于 `VLM_MAX_IMAGE_BYTES`（默认 5 MB）、未启用VLM | `message` 写跳过原因 |
+| `failure` | 必需字段缺失、JSON 修复后仍不合法、VLM/EXTRACT role 未配置而对应模态被启用、模型调用异常 | `message` 写诊断 |
+
+补充：
+
+- `analyze_time` 是 epoch 秒，每个 status 都有；
+- `message` 在 `status="success"` 时**恒为空串**，便于过滤；
+- 对已启用模态的 item，每次 `analyze_multimodal` 都会重新计算，并用本次结果覆盖已有的 `llm_analyze_result`（无论原先是 `success`、`skipped` 还是 `failure`）。这样修正 VLM/EXTRACT 配置后可以直接重试，无须手动清理旧 sidecar 结果。LLM 调用仍会走 analysis cache：如果 cache key 命中，不会再次请求 provider，语义字段通常保持一致，但 `analyze_time` 等运行时字段会被重写。只有 cache miss，例如有效 role 模型 / binding / host、prompt 输入或图片元数据变化后，保存内容才可能与上次不同。
+
+图形 `type` 受 12 项枚举约束（见 [`IMAGE_TYPE_ENUM`](../lightrag/prompt_multimodal.py)：`Photo / Illustration / Screenshot / Icon / Chart / Table / Infographic / Flowchart / Chat Log / Wireframe / Texture / Other`）；模型若返回枚举外的值，会被规整成 `Other` 而不是失败。

docs/LightRAGSidecarFormat.md

@@ -0,0 +1,402 @@
+# LightRAG Sidecar File Format Specification
+
+This document describes the **LightRAG Sidecar** file format that content parsing engines output. When LightRAG uses multimodal-capable content parsing engines such as native/mineru/docling to extract file content, it splits "body text + multimodal objects + parsing metadata" into a `*.parsed/` directory. Each JSON / JSONL file in that directory is collectively called a **sidecar** file. Sidecars are the only reliable source of truth for the subsequent pipeline (multimodal analysis → multimodal chunk construction → entity extraction → cache cleanup on document deletion). The sidecar format is LightRAG's built-in universal file interchange format; new multimodal content extraction engines must follow this format. The purpose of publicly documenting the **LightRAG Sidecar** format is to make it convenient for community developers to write their own content parsing engines.
+
+## 1. Overview
+
+| Concern | File | Contents | Notes |
+|---|---|---|---|
+| Main file | `<doc>.blocks.jsonl` | Stores block body | Concatenating the `content` fields of all blocks reconstructs the complete original text |
+| Drawing objects | `<doc>.drawings.json` | Drawing objects extracted from the file | Sent to a VLM for analysis; analysis results are written back |
+| Table objects | `<doc>.tables.json` | Table objects extracted from the file | Sent to an LLM for analysis; analysis results are written back |
+| Equation objects | `<doc>.equations.json` | Equation objects extracted from the file | Sent to an LLM for analysis; analysis results are written back |
+| Original image assets | `<doc>.blocks.assets/` | Original image files extracted from the document | Sent to a VLM for image analysis |
+
+Design intent of sidecars:
+
+- During the parsing stage, the content extraction engine (native/mineru/docling) is **only** responsible for generating "objective" fields such as `blockid / heading / content / surrounding`;
+- During the multimodal analysis stage (`analyze_multimodal`), the analysis result dict `llm_analyze_result` is written by LightRAG and may be appended or overwritten; parsers should not pre-populate it.
+
+## 2. Directory Layout
+
+```
+inputs/space1/__parsed__/<canonical filename>.parsed/
+├── <canonical filename>.blocks.jsonl        body block sequence + document-level meta (first line)
+├── <canonical filename>.drawings.json       drawing sidecar (dict container, key = drawing id)
+├── <canonical filename>.tables.json         table sidecar
+├── <canonical filename>.equations.json      equation sidecar
+└── <canonical filename>.blocks.assets/      original asset directory (image files referenced by drawings.json live here)
+    ├── image1.wmf
+    ├── image2.wmf
+    ├── image3.wmf
+    ├── image4.png
+    ├── image5.png
+    ├── image6.png
+    └── image7.emf
+```
+
+## 3. blocks.jsonl
+
+`blocks.jsonl` is JSON serialized line by line. The **first line has `type="meta"`**; every subsequent line is a content block with `type="content"`.
+
+### 3.1 meta line example
+
+```json
+{
+  "type": "meta",
+  "format": "lightrag",
+  "version": "1.0",
+  "document_name": "m012-manual.docx",
+  "document_format": "docx",
+  "document_hash": "sha256:4840...3f9543d9db0822d2d59",
+  "table_file": true,
+  "equation_file": true,
+  "drawing_file": true,
+  "asset_dir": true,
+  "blocks": 39,
+  "doc_id": "doc-f1bee60173d067d88595c00e7d9b0ce5",
+  "parse_engine": "native",
+  "parse_time": "2026-05-13T18:42:25.943490+00:00",
+  "doc_title": "m012-manual"
+}
+```
+
+| Field | Type | Description |
+|---|---|---|
+| `type` | `"meta"` | Line type, fixed value, sanity check |
+| `format` | `"lightrag"` | Sidecar major version family identifier |
+| `version` | `str` | Sidecar schema version |
+| `document_name` | `str` | Canonical filename (with extension, without processing hints) |
+| `document_format` | `str` | File format (currently expressed as the file extension) |
+| `document_hash` | `"sha256:<hex>"` | Sidecar body fingerprint, defined as `SHA-256(merged_text)`, where `merged_text` is the concatenation of all non-empty content lines' `content` fields joined by `"\n\n"`. Used by external consumers to quickly determine whether two `.parsed/` directories share the same source (without line-by-line body comparison), and serves as a self-describing content checksum for the sidecar file. Note: the LightRAG ingestion pipeline itself does not read this field; cross-document deduplication is handled separately by `doc_status.content_hash`. |
+| `table_file` / `equation_file` / `drawing_file` | `bool` | Whether the corresponding sidecar files exist (when true, the corresponding file must exist) |
+| `asset_dir` | `bool` | Whether the `blocks.assets` asset directory exists |
+| `split_option` | `object` | Optional. Free-form metadata the parsing engine records about itself (e.g. `engine_version`, engine-specific extras). Omitted entirely when the engine recorded nothing (the common native/markdown case). Chunking is NOT performed at the parse stage — it is the downstream chunker's responsibility — so this field never carries chunking parameters. |
+| `blocks` | `int` | Number of content lines (excluding meta) |
+| `doc_id` | `"doc-<md5>"` | Global document ID. Sidecar item IDs (`im-/tb-/eq-`) use the hash portion of `doc_id` with the `doc-` prefix removed, in order to shorten the placeholder tags embedded in body text. |
+| `parse_engine` | `str` | Parsing engine `native/mineru/docling/legacy` |
+| `parse_time` | `str` | Parse completion time; format: ISO-8601 UTC |
+| `doc_title` | `str` | Document title (usually the first H1); optional |
+| `doc_summary` | `str` | Document summary; optional |
+| `doc_attributes` | `object` | Document extended attributes object; optional |
+| `bbox_attributes` | `object` | Global bbox position attributes; see [§8](#8-positions) |
+
+> LightRAG requires that filenames (`document_name`) be unique within the same workspace (knowledge base).
+
+### 3.2 content line
+
+Each content line is the minimum addressable unit of an original document "block" and contains at least:
+
+```json
+{
+  "type": "content",
+  "blockid": "652e5de55805d1d449b400c0d4a95ca8",
+  "format": "plain_text",
+  "content": "# 1 Product Purpose and Functions\nThe MI012 module is used to support the oxygen-supply and anti-gravity control function of the oxygen-supply and anti-gravity regulator...",
+  "heading": "1 Product Purpose and Functions",
+  "parent_headings": [],
+  "level": 1,
+  "session_type": "body",
+  "table_slice": "none",
+  "positions": [
+    {
+      "type": "paraid",
+      "range": ["5EA4577A", "6555DDCB"]
+    }
+  ]
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `type` | `"content"` |
+| `blockid` | Globally unique Block ID |
+| `format` | Content form, currently fixed to `"plain_text"` |
+| `content` | Text content; **equations and images appear as placeholder tags here, tables appear as JSON or HTML wrapped in table tags** (see §3.3). The heading line is rendered with a markdown `#` prefix (plus a space) matching `level`: level 1 → one `#`, level 2 → two `#`, …, capped at 6 (a level ≥ 7 heading still renders `######`). If the source heading text already begins with a markdown prefix (1–6 `#` followed by a space), it is kept verbatim and not prefixed again. Note the `heading` field itself stays clean (no `#`). |
+| `heading` | The top-most-level heading of the section containing this content. When `heading` is real, it should also appear at the beginning of `content` as the markdown-rendered heading line. **Every recognized heading starts its own block**: if a heading is immediately followed by body text, that body is merged into the same block (content = markdown-rendered heading line + body); if a heading has no following body (e.g., it is immediately followed by a heading at the next level), it still becomes a standalone block whose content is just the markdown-rendered heading line. This ensures that concatenating the `content` fields of all blocks still reconstructs the complete original text without overlap. |
+| `parent_headings` | String array: the top-down list of ancestor headings, excluding the current `heading` |
+| `level` | Integer: the level of `heading` in the document outline (`1` = H1 / first-level heading; `0` means no heading) |
+| `session_type` | The region the block belongs to: `body` `preface` `TOC` `references` `appendix` |
+| `table_slice` | Optional reserved field; indicates whether the block contains only a slice of a table. The current analysis engines do not split long tables, so this field is fixed to `"none"` (meaning the table will not be sliced) |
+| `table_header` | Optional reserved field; when the current block is a table slice, this holds the recognized table header. Currently unused. |
+| `positions` | Array of `position` objects: identifies the layout position of the text block; when the text block comes from multiple positions in the layout, multiple `position` objects appear. See [§8](#8-positions) |
+
+> - blockid computation: `md5(doc_id + ":" + block_index + ":" + heading + ":" + content)`. Chunks produced by chunking strategies record the blockid for tracing the chunk back to its location in the sidecar.
+> - The chunking strategies `F` / `R` / `V` that ignore document section structure operate on the concatenated `content` fields. Therefore, concatenating the `content` fields of all blocks must form the complete document content — no content missing, no content overlapping.
+
+### 3.3 Inline placeholder tags inside content
+
+To let the P chunking strategy split body text without breaking multimodal objects, three XML-style placeholder tags are used inside `content`:
+
+| Tag | Meaning | Tag attributes |
+|---|---|---|
+| `<table id="tb-…" format="json">…</table>` | Table placeholder; the body is the raw table JSON / HTML | `id` points to the corresponding item in `tables.json`; `format` ∈ `json` / `html` |
+| `<drawing id="im-…" format="png" path="…" src="…" caption="…" />` | Self-closing drawing placeholder | `id` points to `drawings.json`; `path` is relative to the `*.parsed/` directory; `src` is the reference name in the original document |
+| `<equation id="eq-…" format="latex" caption="…">…</equation>` | Equation placeholder | Inline equations also use `<equation format="latex">`, but **without** `id`, and are not written to the sidecar; only block equations (occupying one or more entire lines) carry an `id` |
+
+When the text is fed to the LLM during entity/relation extraction, internal attributes such as `id / path / src` are stripped, but key attributes (`format / caption`) are preserved. The goal is to avoid extracting entities that are invisible in the article and injecting too much noise into the extraction results.
+
+### 3.4 Correspondence between blockid and chunk sidecar.refs
+
+When a sidecar file exists, the chunking strategies attach `sidecar = {"type": "block", "id": <primary source blockid>, "refs": [{"type": "block", "id": <blockid>}, …]}` to each output chunk, where:
+
+- Unmerged chunk → `sidecar.refs` has only one element, equal to the `blockid` of the blocks.jsonl line the chunk came from;
+- Chunk merged in LevelMerge → `refs` preserves the order of all source `blockid`s (deduplicated);
+- Sub-chunks after hard fallback split → share the parent chunk's `sidecar`.
+
+This linkage is the basis for document-level traceability (chunk ↔ block ↔ original paragraph paraId).
+
+## 4. drawings.json
+
+The top level is a dict container of the form `{"version": "1.0", "drawings": { <id>: <item>, … }}`, **keyed by the `id` field** for lookup by id. Each item looks like:
+
+```json
+{
+  "id": "im-f1bee60173d067d88595c00e7d9b0ce5-0004",
+  "blockid": "2f52b70839d13a936d97955916820147",
+  "heading": "2.3 Structural Dimensions and Weight",
+  "parent_headings": ["2 Product Description"],
+  "format": "png",
+  "path": "m012-manual.blocks.assets/image4.png",
+  "src": "",
+  "caption": "",
+  "footnotes": [],
+  "extras": {
+    "ocr_texts": "First OCR paragraph inside the image\n\nSecond OCR paragraph inside the image",
+    "ocr_texts_count": 2
+  },
+  "surrounding": {
+    "leading": "2.3 Structural Dimensions and Weight\nDimensional and weight requirements are as follows:\na) Outer dimensions length: <drawing …",
+    "trailing": "\nFigure 1  Outer dimension schematic\nb) Weight does not exceed 0.85 kg.\nc) Test result: measured circuit noise Vpp=1.526 mV…"
+  },
+  "llm_analyze_result": {
+    "name": "Product outer-dimension engineering drawing",
+    "type": "Illustration",
+    "description": "This drawing is a schematic of the product's outer dimensions, presenting three views of an electronic device or power module design…",
+    "analyze_time": 1778697752,
+    "status": "success",
+    "message": ""
+  },
+  "llm_cache_list": [
+    "default:analysis:fcf4c4f88227ee1c1bf0ed4394039e37"
+  ]
+}
+```
+
+| Field | Description |
+|---|---|
+| `id` | Form `im-<doc_hash>-<NNNN>` (`doc_hash` is the 32-character md5 portion of `doc_id` with the `doc-` prefix removed) |
+| `blockid` | Points to the content line that produced this drawing |
+| `heading` | The section heading the drawing belongs to |
+| `parent_headings` | String array: the top-down list of ancestor headings, excluding the current `heading` (mirrors the `blocks.jsonl` field of the same name on the block this drawing belongs to) |
+| `format` | Original extension (no dot): `png` / `jpeg` / `gif` / `webp` / `wmf` / `emf` / … |
+| `path` | Resource path relative to the `*.parsed/` directory; **always** points to a file inside `*.blocks.assets/` |
+| `src` | The reference alias of the drawing in the original document (empty in most cases) |
+| `caption` | Visible caption (the parser may leave it empty) |
+| `footnotes` | List of footnote strings |
+| `surrounding` | Context object: see [§7](#7-surrounding) |
+| `self_ref` | String, optional; an object reference from the original parsing engine output (e.g., Docling JSON Pointer `#/pictures/3`, or MinerU `content_list.json#/23`), used to look up the original object in the parsing artifacts (page position, original structure, etc.) when tracing back. Not output by `native` and other engines that do not provide this field. |
+| `extras` | Object, optional; engine-specific bypass fields (such as OCR text contained inside the image, etc.). Not part of spec validation; downstream consumers should not rely on specific keys. |
+| `llm_analyze_result` | Modal analysis result object: see [§9](#9-llm_analyze_result) (will later be injected into the multimodal text block) |
+| `llm_cache_list` | LLM cache list for modal analysis (will later be injected into the multimodal text block) |
+
+Common drawing-specific keys inside `extras`:
+
+| Key | Description |
+|---|---|
+| `ocr_texts` | String, optional; OCR text inside the drawing object, with multiple paragraphs concatenated by blank lines (`\n\n`). Only written when the parsing engine explicitly attaches OCR text under this drawing's children; caption / footnote do not enter this field. |
+| `ocr_texts_count` | Integer, optional; number of non-empty OCR paragraphs written into `ocr_texts`. |
+
+**Only raster formats supported by drawings (png / jpeg / gif / webp) enter VLM analysis**; other formats (wmf / emf / svg, etc.) get `llm_analyze_result.status="skipped"`, no multimodal chunk is generated downstream, and document processing continues. Images larger than the size specified by the environment variable `VLM_MAX_IMAGE_BYTES` likewise will not enter VLM analysis.
+
+> Information such as image size and DPI is uniformly placed in the `extras` object; do not introduce undeclared fields (like `image` / `img_path`, etc.) at the item top level. tables / equations follow the same `extras` convention. `self_ref` is a top-level optional field declared by the spec and does not belong to `extras`.
+
+## 5. tables.json
+
+The top level is a dict container of the form `{"version": "1.0", "tables": { <id>: <item>, ... }}`, **keyed by the `id` field** for lookup by id. Each item looks like:
+
+```json
+{
+  "id": "tb-f1bee60173d067d88595c00e7d9b0ce5-0007",
+  "blockid": "3f33897b5e105d254addc655f1efbf8c",
+  "heading": "2.4.4 Temperature-Humidity-Altitude (run with the system)",
+  "parent_headings": ["2 Product Description", "2.4 Environmental Adaptability"],
+  "dimension": [16, 8],
+  "format": "json",
+  "content": "[[\"Step\", \"Temperature (°C)\", \"Altitude (m)\", \"Relative humidity\", \"Time (min)\", \"Auxiliary cooling\", \"System power\", \"Functional/performance check\"],…",
+  "caption": "",
+  "footnotes": [],
+  "table_header": "[[\"Step\", \"Temperature (°C)\", \"Altitude (m)\", \"Relative humidity\", \"Time (min)\", \"Auxiliary cooling\", \"System power\", \"Functional/performance check\"]]"
+  "surrounding": {
+    "leading": "2.4.4 Temperature-Humidity-Altitude (run with the system)\nThe product shall withstand the combined temperature, humidity, and altitude environment during mission execution…",
+    "trailing": "\nNote: the above steps are repeated for 10 cycles. a) Finished product and accessories reach thermal stability or 240 min, whichever is longer; b) Finished product and accessories reach thermal stability or 120 min, whichever is longer.…"
+  },
+  "llm_analyze_result": {
+    "name": "Document management metadata table",
+    "description": "This is a document management information table used to record basic metadata and version control information for a technical document …",
+    "analyze_time": 1778697759,
+    "status": "success",
+    "message": ""
+  },
+  "llm_cache_list": [
+    "default:analysis:b316aacd40fdca0cb56430870bb89a62"
+  ]
+}
+```
+
+The `blockid` / `heading` / `parent_headings` / `surrounding` / `llm_analyze_result` fields of tables.json have the same meaning as in drawings.json. Different or newly added fields are described below:
+
+| Field | Description |
+|---|---|
+| `id` | Form `tb-<doc_hash>-<NNNN>` (`doc_hash` is the 32-character md5 portion of `doc_id` with the `doc-` prefix removed) |
+| `dimension` | Integer array: `[num_rows, num_cols]`, including header rows |
+| `format` | `"json"` (2D array) or `"html"` (payload `<table>…</table>` fragment including the opening and closing tags) |
+| `content` | String: the table body, structured according to `format`; this is the string actually used by the downstream multimodal chunk. |
+| `table_header` | String, optional; the recognized row(s) treated as the table header |
+| `self_ref` | Optional; object reference from the original parsing engine output (e.g., Docling JSON Pointer `#/tables/2`, or MinerU `content_list.json#/31`), used to look up the original artifact when tracing back |
+
+During the modal analysis stage, when the length of the `content` field exceeds the LLM's context window, the table content is mechanically truncated before being fed to the model.
+
+## 6. equations.json
+
+The top level is a dict container of the form `{"version": "1.0", "equations": { <id>: <item>, ... }}`, **keyed by the `id` field** for lookup by id. Each item looks like:
+
+```json
+{
+  "id": "eq-f1bee60173d067d88595c00e7d9b0ce5-0001",
+  "blockid": "2f52b70839d13a936d97955916820147",
+  "heading": "2.3 Structural Dimensions and Weight",
+  "parent_headings": ["2 Product Description"],
+  "format": "latex",
+  "content": "C=2∗\\frac{P∗T}{\\left( {V}_{H}^{2}−{V}_{L}^{2} \\right)∗η}",
+  "caption": "",
+  "footnotes": [],
+  "surrounding": {
+    "leading": "2.3 Structural Dimensions and Weight\nDimensional and weight requirements are as follows:\n …",
+    "trailing": "\nwhere P is the power maintained during power abnormalities 28 W, T is the desired energy-storage time, V<sub>H</sub> is before capacitor discharge…"
+  },
+  "llm_analyze_result": {
+    "name": "Capacitor energy-storage time calculation formula",
+    "description": "This formula calculates the capacitor energy storage value required to maintain normal system operation during power abnormality …",
+    "analyze_time": 1778697783,
+    "status": "success",
+    "message": "",
+    "equation": "C=2\\cdot\\frac{P\\cdot T}{(V_{H}^{2}-V_{L}^{2})\\cdot\\eta}"
+  },
+  "llm_cache_list": [
+    "default:analysis:fcf4c4f88227ee1c1bf0ed4394039e37"
+  ]
+}
+```
+
+The `blockid` / `heading` / `parent_headings` / `surrounding` / `llm_analyze_result` fields of equations.json have the same meaning as in drawings.json. Different or newly added fields are described below:
+
+| Field | Description |
+|---|---|
+| `id` | Form `eq-<doc_hash>-<NNNN>` (`doc_hash` is the 32-character md5 portion of `doc_id` with the `doc-` prefix removed) |
+| `format` | Fixed to `"latex"` |
+| `content` | String: the **raw** LaTeX (possibly containing Unicode operators, outer `\[ \]`); does not include the leading/trailing `$` delimiters; read directly by the modal analysis stage |
+| `self_ref` | Optional; object reference from the original parsing engine output (e.g., Docling JSON Pointer `#/texts/15`, or MinerU `content_list.json#/45`), used to look up the original artifact when tracing back |
+| `llm_analyze_result.equation` | String: the **canonicalized** LaTeX equation output by the LLM (outer `$ / \[ \] / equation` environment, Unicode converted to LaTeX, no leading/trailing `$` delimiters); this is the string actually used by the downstream multimodal chunk. |
+
+During the modal analysis stage, when the length of the `content` field exceeds the LLM's context window, the content is mechanically truncated before being fed to the model. Inline equations (those continuous with the body, as `<equation format="latex">…</equation>`) **are not** saved to equations.json; they remain only in the blocks text without an `id`. The goal is to avoid injecting too much noise into the extraction results.
+
+## 7. surrounding
+
+`surrounding.leading` and `surrounding.trailing` are the analyzable context windows of a sidecar item; their purpose is to provide contextual information about the paragraph containing the image, table, or equation, improving the quality of multimodal analysis. **The surrounding content is automatically injected by LightRAG during the analysis stage; it does not need to be actively written into the sidecar by the document parsing engine.** The generation logic of the surrounding content is as follows:
+
+- Taken from the text of the content line with the same `blockid`, split at the position of the multimodal placeholder tag;
+- The token limit on each side is controlled by the environment variables `SURROUNDING_LEADING_MAX_TOKENS` / `SURROUNDING_TRAILING_MAX_TOKENS` (default `2000`, can be tuned independently); truncated by tokenizer, preferring to retain sentences close to the target;
+- The text preserves placeholder tags of **other multimodal objects on the same line**, allowing the model to perceive context such as "after Figure 1 there is also Equation 1"; but internal parser identifiers (`id` / `path` / `src` / `refid`) have been stripped by `strip_internal_multimodal_markup_for_extraction` — consistent with chunk content cleanup before entity extraction, to avoid noise entering the VLM/LLM prompt. Specific cleanup rules:
+  - `<drawing id="im-…" path="…" src="…" caption="Fig 1" />` → `<drawing caption="Fig 1" />`; **drawings without a caption are removed entirely** (the tag carries no model-visible information anymore);
+  - `<table id="tb-…" format="json" caption="…">rows</table>` → `<table format="json" caption="…">rows</table>`;
+  - `<equation id="eq-…" format="latex">body</equation>` → `<equation format="latex">body</equation>`;
+  - `<cite type="table" refid="tb-…">Table 1</cite>` → `<cite type="table">Table 1</cite>`; `<cite type="equation" refid="eq-…">Equation 2</cite>` → `<cite type="equation">Equation 2</cite>`. Only the `refid` attribute is removed; the `<cite type="…">…</cite>` wrapper is preserved — letting the VLM/LLM recognize "this is a reference to another table/equation" rather than ordinary text, while hiding the parser-internal id that the LLM cannot see.
+    - Exception: surrounding of the `tables.json` type first goes through `remove_table_tags` before stripping, removing all `<cite type="table">` blocks entirely (when analyzing the target table, we don't want to be distracted by dangling references to other tables);
+- Cleanup happens **before** token-budget truncation: the token count is computed on "what the LLM actually sees", and truncation does not land inside an uncleaned `id="…"` attribute, avoiding broken tag structure;
+- When the target object itself sits at the start / end of the block, the corresponding side is `""` instead of `"n/a"` (when assembling the prompt, the empty string is later displayed as `n/a`);
+- `enrich_sidecars_with_surrounding` is idempotent: each `analyze_multimodal` entry point recomputes and overwrites `surrounding`, so after changing `SURROUNDING_LEADING_MAX_TOKENS` / `SURROUNDING_TRAILING_MAX_TOKENS` there is no need to manually clean the sidecar — just re-run multimodal analysis and `surrounding` will be rewritten under the new budget.
+
+## 8. positions
+
+`positions` is an array of objects that identifies which piece of text in the file the `blockid` content comes from, allowing the original content to be located and displayed in the source file during content traceability. When the content of a `blockid` is composed of several columns from the layout, multiple `position` objects appear, with each `position` object corresponding to one layout box or column. To accommodate different document formats' content positioning approaches, the system supports the following types of `position` object.
+
+`position` objects have multiple types, and the `type` field determines its type:
+
+* paraid
+
+Applicable to docx-format files; locates content by `paragraph id` (paraid). The `range` field specifies the start and end `paragraph id`s; `charspan` is an optional field specifying that the content starts at character m and ends at character n of the paragraph. When `charspan` is not provided, the `blockid` covers the entire content of the start and end paragraphs. Example:
+
+```
+"positions": [
+{
+    "type": "paraid",
+    "range": ["5EA4577A", "6555DDCB"]
+    "charspan": [10,999]
+}]
+```
+
+* bbox
+
+Applicable to PDF-like files; identifies the original position of the content via a rectangle on the page. bbox supports the following fields:
+
+```
+origin: Which position the rectangle coordinates are relative to on the page (optional, defaults to LEFTTOP; another option is LEFTBOTTOM)
+max: Maximum length and width of the page layout; coordinates are normalized by this value for accurate position display (optional; empty means coordinates are computed by the image's pixel grid)
+anchor: Page number, as a string, supporting non-Arabic page numbers such as Roman numerals
+range: Rectangle coordinate array [h1, w1, h2, w2], e.g., [174, 155, 818, 333]
+charspan: Content starts at character m and ends at character n of the anchored paragraph (optional)
+```
+
+The `bbox_attributes` field of the `meta` line in `blocks.jsonl` holds global bbox settings, avoiding repeating the same content in every `content` line's `positions` object. A typical `positions` object example:
+
+```
+"positions": [
+{
+    "type": "bbox",
+    "anchor": "ii"
+    "range": [174, 155, 818, 333]
+    "charspan": [10, 999]
+}]
+```
+
+* heading
+
+Applicable to Markdown-like files; locates content by heading. `anchor` is the starting heading (for handling duplicated headings, refer to the Markdown anchor specification); `charspan` is an optional field specifying that the content starts at character m and ends at character n of the paragraph. When `charspan` is not provided, the `blockid` covers the entire content of the start and end paragraphs.
+
+```
+"positions": [
+{
+    "type": "heading",
+    "anchor": "ii"
+    "range": [174, 155, 818, 333]
+    "charspan": [10, 999]
+}]
+```
+
+* absolute
+
+Applicable to text-like files; locates content by absolute character position. `charspan` specifies that the content starts at character m and ends at character n.
+
+```
+"positions": [
+{
+    "charspan": [10, 999]
+}]
+```
+
+## 9. `llm_analyze_result`
+
+| `status` | Trigger scenario | Field description |
+|---|---|---|
+| `success` | The model returns valid JSON and all required fields are present | Drawing: `name / type / description`; Table: `name / description`; Equation: `name / description / equation` |
+| `skipped` | Multimodal analysis was deliberately skipped: image format unsupported, pixels < `VLM_MIN_IMAGE_PIXEL` (default 32 px), larger than `VLM_MAX_IMAGE_BYTES` (default 5 MB), or VLM not enabled | `message` records the skip reason |
+| `failure` | Required fields missing, JSON still invalid after repair, the VLM/EXTRACT role is not configured while the corresponding modality is enabled, or the model invocation throws an exception | `message` records the diagnostic |
+
+Additional notes:
+
+- `analyze_time` is epoch seconds and is present for every status;
+- `message` is **always an empty string** when `status="success"`, making filtering convenient;
+- Items for enabled modalities are recomputed on each `analyze_multimodal` run, and the current run overwrites any prior `llm_analyze_result` (`success`, `skipped`, or `failure`). This allows operators to fix VLM/EXTRACT configuration and retry without manually clearing stale sidecar results. LLM calls still use the analysis cache: if the cache key matches, the provider is not called and semantic fields usually remain the same, though runtime fields such as `analyze_time` are rewritten. A cache miss, for example after changing the effective role model/binding/host, prompt inputs, or image metadata, can produce different saved content.
+
+Drawing `type` is constrained to a 12-value enum (see [`IMAGE_TYPE_ENUM`](../lightrag/prompt_multimodal.py): `Photo / Illustration / Screenshot / Icon / Chart / Table / Infographic / Flowchart / Chat Log / Wireframe / Texture / Other`); values returned by the model outside the enum are normalized to `Other` rather than failing.

docs/MilvusConfigurationGuide.md

@@ -0,0 +1,197 @@
+# Milvus Configuration via vector_db_storage_cls_kwargs
+
+## Overview
+
+Milvus index parameters can be configured through `vector_db_storage_cls_kwargs`, which is the **recommended approach** for framework integration scenarios (e.g., when using RAGAnything or other frameworks built on top of LightRAG).
+
+## Why Use vector_db_storage_cls_kwargs?
+
+✅ **Framework Integration**: Allows configuration to be passed through framework layers without environment variable changes
+✅ **Programmatic Configuration**: Set parameters in code rather than relying on environment variables
+✅ **Dynamic Configuration**: Different configurations for different RAG instances
+✅ **Clean API**: All parameters passed in one place during initialization
+
+## Supported Parameters
+
+All 11 MilvusIndexConfig parameters can be configured via `vector_db_storage_cls_kwargs`:
+
+### Base Configuration
+- `index_type`: Index type (AUTOINDEX, HNSW, HNSW_SQ, IVF_FLAT, etc.)
+- `metric_type`: Distance metric (COSINE, L2, IP)
+
+### HNSW Parameters
+- `hnsw_m`: Number of connections per layer (2-2048, default: 16)
+- `hnsw_ef_construction`: Size of dynamic candidate list during construction (default: 360)
+- `hnsw_ef`: Size of dynamic candidate list during search (default: 200)
+
+### HNSW_SQ Parameters (requires Milvus 2.6.8+)
+- `sq_type`: Quantization type (SQ4U, SQ6, SQ8, BF16, FP16, default: SQ8)
+- `sq_refine`: Enable refinement (default: False)
+- `sq_refine_type`: Refinement type (SQ6, SQ8, BF16, FP16, FP32, default: FP32)
+- `sq_refine_k`: Number of candidates to refine (default: 10)
+
+### IVF Parameters
+- `ivf_nlist`: Number of cluster units (1-65536, default: 1024)
+- `ivf_nprobe`: Number of units to query (default: 16)
+
+## Configuration Priority
+
+Configuration is resolved in the following order:
+1. **Parameters passed via vector_db_storage_cls_kwargs** (highest priority)
+2. Environment variables (MILVUS_INDEX_TYPE, etc.)
+3. Default values
+
+## Usage Examples
+
+### Basic Configuration
+
+```python
+from lightrag import LightRAG
+
+rag = LightRAG(
+    working_dir="./demo",
+    vector_storage="MilvusVectorDBStorage",
+    vector_db_storage_cls_kwargs={
+        "cosine_better_than_threshold": 0.2,
+        "index_type": "HNSW",
+        "metric_type": "COSINE",
+        "hnsw_m": 32,
+        "hnsw_ef_construction": 256,
+        "hnsw_ef": 150,
+    }
+)
+```
+
+### RAGAnything Framework Integration
+
+```python
+# In RAGAnything framework code:
+def create_lightrag_instance(user_config):
+    """Create LightRAG instance with user-provided Milvus configuration"""
+
+    # User configuration from RAGAnything
+    milvus_config = {
+        "cosine_better_than_threshold": user_config.get("threshold", 0.2),
+        "index_type": user_config.get("index_type", "HNSW"),
+        "hnsw_m": user_config.get("hnsw_m", 32),
+        # ... other parameters
+    }
+
+    # Pass configuration to LightRAG
+    rag = LightRAG(
+        working_dir=user_config["working_dir"],
+        vector_storage="MilvusVectorDBStorage",
+        vector_db_storage_cls_kwargs=milvus_config,
+    )
+
+    return rag
+```
+
+### Advanced Configuration with HNSW_SQ
+
+```python
+rag = LightRAG(
+    working_dir="./demo",
+    vector_storage="MilvusVectorDBStorage",
+    vector_db_storage_cls_kwargs={
+        "cosine_better_than_threshold": 0.2,
+        "index_type": "HNSW_SQ",  # Requires Milvus 2.6.8+
+        "metric_type": "COSINE",
+        "hnsw_m": 48,
+        "hnsw_ef_construction": 400,
+        "hnsw_ef": 200,
+        "sq_type": "SQ8",
+        "sq_refine": True,
+        "sq_refine_type": "FP32",
+        "sq_refine_k": 20,
+    }
+)
+```
+
+### IVF Configuration
+
+```python
+rag = LightRAG(
+    working_dir="./demo",
+    vector_storage="MilvusVectorDBStorage",
+    vector_db_storage_cls_kwargs={
+        "cosine_better_than_threshold": 0.2,
+        "index_type": "IVF_FLAT",
+        "metric_type": "L2",
+        "ivf_nlist": 2048,
+        "ivf_nprobe": 32,
+    }
+)
+```
+
+## Implementation Details
+
+### How It Works
+
+1. When `MilvusVectorDBStorage.__post_init__()` is called:
+   ```python
+   kwargs = self.global_config.get("vector_db_storage_cls_kwargs", {})
+   index_config_keys = MilvusIndexConfig.get_config_field_names()
+   index_config_params = {
+       k: v for k, v in kwargs.items() if k in index_config_keys
+   }
+   self.index_config = MilvusIndexConfig(**index_config_params)
+   ```
+
+2. `MilvusIndexConfig.get_config_field_names()` dynamically extracts all valid parameter names from the dataclass
+3. Only valid Milvus index parameters are extracted from kwargs
+4. Parameters are passed to `MilvusIndexConfig` which applies defaults and validates them
+5. Environment variables are used as fallback for any parameters not provided in kwargs
+
+### Automatic Synchronization
+
+The implementation uses `MilvusIndexConfig.get_config_field_names()` to dynamically extract valid parameters. This means:
+- ✅ New parameters added to `MilvusIndexConfig` are **automatically recognized**
+- ✅ No need to maintain duplicate parameter lists
+- ✅ Single source of truth for configuration parameters
+
+## Testing
+
+The configuration via `vector_db_storage_cls_kwargs` is thoroughly tested:
+
+```bash
+# Run all kwargs bridge tests
+python -m pytest tests/kg/milvus_impl/test_milvus_kwargs_bridge.py -v
+
+# Test RAGAnything integration scenario specifically
+python -m pytest tests/kg/milvus_impl/test_milvus_kwargs_bridge.py::TestMilvusKwargsParameterBridge::test_raganything_framework_integration_scenario -v
+
+# Test all parameters support
+python -m pytest tests/kg/milvus_impl/test_milvus_kwargs_bridge.py::TestMilvusKwargsParameterBridge::test_all_milvus_parameters_supported_via_kwargs -v
+```
+
+## Examples
+
+See `examples/milvus_kwargs_configuration_demo.py` for a complete working example.
+
+## Backward Compatibility
+
+✅ **100% backward compatible** with existing code
+✅ Environment variable configuration still works
+✅ All existing tests pass
+
+## FAQ
+
+### Q: Can I mix kwargs and environment variables?
+**A:** Yes! Parameters in `vector_db_storage_cls_kwargs` take priority over environment variables.
+
+### Q: What happens to non-Milvus parameters in kwargs?
+**A:** They are ignored. Only valid MilvusIndexConfig parameters are extracted. This allows frameworks to pass their own parameters alongside Milvus configuration.
+
+### Q: Do I need to set environment variables?
+**A:** No! When using `vector_db_storage_cls_kwargs`, environment variables are optional. They serve as fallback values.
+
+### Q: Is this approach recommended for RAGAnything?
+**A:** Yes! This is the **recommended approach** for any framework that builds on top of LightRAG, as it allows clean configuration passing through framework layers.
+
+## References
+
+- Test Suite: `tests/kg/milvus_impl/test_milvus_kwargs_bridge.py`
+- Implementation: `lightrag/kg/milvus_impl.py` (lines 1237-1272)
+- Example: `examples/milvus_kwargs_configuration_demo.py`
+- MilvusIndexConfig: `lightrag/kg/milvus_impl.py` (lines 75-303)

docs/MultiSiteDeployment.md

@@ -0,0 +1,382 @@
+# Single-Server Multi-Site Deployment
+
+This document explains how to run multiple isolated LightRAG instances behind one host using a reverse proxy (nginx, Traefik, Kubernetes Ingress, …), with **one shared WebUI build** reused by every instance.
+
+> Looking for the basic single-instance Docker setup? See [DockerDeployment.md](./DockerDeployment.md). For frontend build
+> mechanics in general, see [FrontendBuildGuide.md](./FrontendBuildGuide.md).
+
+---
+
+## TL;DR
+
+- Set `LIGHTRAG_API_PREFIX` per-instance, on the **backend only**. The WebUI is always mounted at `/webui` (not configurable).
+- Build the WebUI **once**. The same artifacts work under any reverse-proxy prefix.
+- Point your reverse proxy at each backend, stripping the site prefix before forwarding.
+
+```bash
+# One image, two containers, two prefixes — no rebuild.
+docker run -e LIGHTRAG_API_PREFIX=/site01 -p 9621:9621 lightrag:latest
+docker run -e LIGHTRAG_API_PREFIX=/site02 -p 9622:9621 lightrag:latest
+```
+
+---
+
+## Why "build once, deploy many"
+
+Earlier versions of LightRAG baked the site prefix into the JavaScript bundle at build time (via `VITE_API_PREFIX` / `VITE_WEBUI_PREFIX`). Every site that used a different prefix needed its own WebUI build, and reusing a single Docker image across sites required a rebuild step at deploy time. Since the runtime-config-injection refactor:
+
+- **Asset URLs** in `index.html` are emitted as relative paths (`./assets/index-abc.js`). The browser resolves them against the current document URL, so they work under any mount point.
+- **API base URL** and **in-app links** read their prefix from `window.__LIGHTRAG_CONFIG__`, which the FastAPI server injects into `index.html` on each response based on its own `LIGHTRAG_API_PREFIX`.
+
+The result: a single `lightrag/api/webui/` directory (or Docker image) is reusable across any number of sites with no per-site build artifact.
+
+---
+
+## How runtime prefix injection works
+
+Each request for `index.html` goes through `SmartStaticFiles` in `lightrag/api/lightrag_server.py`, which:
+
+1. Reads the static `index.html` produced by `bun run build`.
+2. Looks for the placeholder comment `<!-- __LIGHTRAG_RUNTIME_CONFIG__ -->`.
+3. Replaces it with
+   `<script>window.__LIGHTRAG_CONFIG__ = {"apiPrefix":"…","webuiPrefix":"…"}</script>`,
+   computed from the configured `LIGHTRAG_API_PREFIX` (the in-app `/webui` mount is hardcoded server-side).
+
+Sequence — browser request to a site-prefixed instance:
+
+```
+Browser            nginx                  uvicorn         SmartStaticFiles
+  │                  │                       │                    │
+  │ GET /site01/webui/                       │                    │
+  │─────────────────►│                       │                    │
+  │                  │ GET /webui/  (strips /site01)              │
+  │                  │──────────────────────►│                    │
+  │                  │                       │ get_response("")   │
+  │                  │                       │───────────────────►│
+  │                  │                       │                    │ inject
+  │                  │                       │                    │ window.__LIGHTRAG_CONFIG__
+  │                  │                       │                    │ = { apiPrefix: "/site01",
+  │                  │                       │                    │ webuiPrefix: "/site01/webui/" }
+  │                  │                       │◄───────────────────│
+  │                  │◄──────────────────────│                    │
+  │◄─────────────────│                       │                    │
+  │ index.html with injected runtime config
+```
+
+The SPA reads the injected config via `src/lib/runtimeConfig.ts` and uses
+it for `axios.baseURL`, `fetch()` template strings, the API-docs iframe,
+and in-app links.
+
+---
+
+## One backend variable, that's it
+
+| Variable | Default | Meaning |
+| --- | --- | --- |
+| `LIGHTRAG_API_PREFIX` | `""` | Reverse-proxy mount prefix. The backend accepts both strip and verbatim forwarding — pick whichever fits your proxy stack. Passed to FastAPI as `root_path`. |
+
+The WebUI is always mounted at `/webui` server-side. `window.__LIGHTRAG_CONFIG__.webuiPrefix` is computed as `LIGHTRAG_API_PREFIX + "/webui/"` and injected for the SPA — you do **not** set it yourself.
+
+There are no longer any frontend `VITE_API_PREFIX` / `VITE_WEBUI_PREFIX` variables. Setting them has no effect (they are ignored by the build).
+
+### Forwarding modes: strip and verbatim both work
+
+After setting `LIGHTRAG_API_PREFIX=/site01`, the backend resolves all routes correctly under either forwarding style:
+
+- **Strip** — proxy removes the prefix, backend sees `/webui/` and `/documents/foo`. The nginx example below uses this style.
+- **Verbatim** — proxy forwards the request unchanged, backend sees `/site01/webui/` and `/site01/documents/foo`. The Vite dev flow ([Scenario 2](#scenario-2--simulate-a-site-prefix)) and any non-rewriting proxy use this style.
+
+A small ASGI middleware in `create_app` prepends `root_path` to `scope["path"]` whenever the path does not already include it, so plain Routes and Mount sub-apps (the WebUI's `StaticFiles`) both resolve identically in either mode. You do not need to standardize on one — both coexist on the same backend without configuration toggles.
+
+---
+
+## End-to-end example: two sites behind one nginx
+
+### Instance configuration
+
+`site01.env`:
+```bash
+HOST=0.0.0.0
+PORT=9621
+LIGHTRAG_API_PREFIX=/site01
+WORKING_DIR=/data/site01/storage
+INPUT_DIR=/data/site01/inputs
+LIGHTRAG_API_KEY=site01-secret
+# … LLM / embedding config …
+```
+
+`site02.env`:
+```bash
+HOST=0.0.0.0
+PORT=9621
+LIGHTRAG_API_PREFIX=/site02
+WORKING_DIR=/data/site02/storage
+INPUT_DIR=/data/site02/inputs
+LIGHTRAG_API_KEY=site02-secret
+# … LLM / embedding config …
+```
+
+### docker-compose.yml (one image, two services)
+
+```yaml
+services:
+  site01:
+    image: ghcr.io/hkuds/lightrag:latest
+    env_file: site01.env
+    volumes:
+      - ./data/site01:/data/site01
+    ports:
+      - "127.0.0.1:9621:9621"
+
+  site02:
+    image: ghcr.io/hkuds/lightrag:latest
+    env_file: site02.env
+    volumes:
+      - ./data/site02:/data/site02
+    ports:
+      - "127.0.0.1:9622:9621"
+```
+
+### nginx config
+
+```nginx
+server {
+    listen 443 ssl http2;
+    server_name host.example.com;
+
+    # site01: strips /site01/ before forwarding
+    location /site01/ {
+        proxy_pass http://127.0.0.1:9621/;
+        proxy_set_header X-Forwarded-Prefix /site01;
+        proxy_set_header Host $host;
+        proxy_http_version 1.1;
+        proxy_set_header Connection "";
+    }
+
+    # site02: strips /site02/ before forwarding
+    location /site02/ {
+        proxy_pass http://127.0.0.1:9622/;
+        proxy_set_header X-Forwarded-Prefix /site02;
+        proxy_set_header Host $host;
+        proxy_http_version 1.1;
+        proxy_set_header Connection "";
+    }
+}
+```
+
+Browsing `https://host.example.com/site01/webui/` shows site01's WebUI; `https://host.example.com/site02/webui/` shows site02's. The same Docker image serves both — no per-site build artifact, no rebuild on prefix changes.
+
+### What each layer sees
+
+| Layer | site01 GET /webui/ |
+| --- | --- |
+| Browser address bar | `https://host.example.com/site01/webui/` |
+| nginx receives | `/site01/webui/` |
+| nginx forwards | `/webui/` |
+| FastAPI `root_path` | `/site01` |
+| `app.mount` resolves | `/webui/` |
+| Injected `apiPrefix` | `/site01` |
+| Injected `webuiPrefix` | `/site01/webui/` |
+| Asset URLs in HTML | `./assets/index-abc.js` (resolves to `https://host.example.com/site01/webui/assets/index-abc.js`) |
+
+---
+
+## Single-image Docker recipe
+
+The `Dockerfile` builds the WebUI once, with no prefix:
+
+```dockerfile
+FROM oven/bun:1 AS webui-build
+WORKDIR /src/lightrag_webui
+COPY lightrag_webui/package.json lightrag_webui/bun.lock ./
+RUN bun install --frozen-lockfile
+COPY lightrag_webui/ ./
+COPY lightrag/api/webui/.gitkeep /src/lightrag/api/webui/.gitkeep
+RUN bun run build
+
+FROM python:3.11-slim
+COPY --from=webui-build /src/lightrag/api/webui /app/lightrag/api/webui
+# … rest of the image …
+```
+
+Run any number of containers from the same image, each with its own prefix:
+
+```bash
+# Plain single-instance, no prefix.
+docker run --rm -p 9621:9621 lightrag:latest
+
+# Same image, different prefixes — runtime decides.
+docker run --rm -e LIGHTRAG_API_PREFIX=/site01 -p 9621:9621 lightrag:latest
+docker run --rm -e LIGHTRAG_API_PREFIX=/site02 -p 9622:9621 lightrag:latest
+```
+
+### Kubernetes Ingress equivalent
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: lightrag-multisite
+  annotations:
+    nginx.ingress.kubernetes.io/rewrite-target: /$2
+spec:
+  rules:
+  - host: host.example.com
+    http:
+      paths:
+      - path: /site01(/|$)(.*)
+        pathType: ImplementationSpecific
+        backend:
+          service:
+            name: lightrag-site01
+            port: { number: 9621 }
+      - path: /site02(/|$)(.*)
+        pathType: ImplementationSpecific
+        backend:
+          service:
+            name: lightrag-site02
+            port: { number: 9621 }
+```
+
+Backends still set `LIGHTRAG_API_PREFIX=/site01` / `=/site02`.
+
+---
+
+## Local development with `bun run dev`
+
+> **Always open `http://localhost:5173/` — root path, no `/webui`, no `/site01` — regardless of which scenario below you're in.**
+>
+> Vite's dev server serves the SPA at its own root (`/`) no matter what prefix you configure. `VITE_DEV_API_PREFIX` only affects how the SPA composes API URLs *after* the page is loaded, and which paths the dev proxy intercepts; it does **not** change the URL you type in the address bar. Trying to access `localhost:5173/site01/webui/` works (Vite's SPA fallback returns the same `index.html`), but it's not the canonical entry point and only differs cosmetically in the address bar.
+>
+> This is the deliberate consequence of `base: './'` in [`vite.config.ts`](../lightrag_webui/vite.config.ts) — the same setting that makes one production build reusable across any number of reverse-proxy mount points. Tying the dev URL to a prefix would force the build to bake the prefix back in.
+
+The dev server mirrors production injection: it serves `index.html` via the same `transformIndexHtml` mechanism the FastAPI server uses at request time, so the SPA reads `window.__LIGHTRAG_CONFIG__` in dev exactly the way it does in prod. Only **two** environment variables matter:
+
+| Variable | Purpose | Where it lives |
+| --- | --- | --- |
+| `VITE_BACKEND_URL` | Where the dev server forwards proxied API calls. | `lightrag_webui/.env*` |
+| `VITE_DEV_API_PREFIX` | Prefix to **simulate** (matches the backend LIGHTRAG_API_PREFIX`). Empty → no prefix. | `lightrag_webui/.env*` |
+
+`VITE_DEV_API_PREFIX` injects `apiPrefix` into `window.__LIGHTRAG_CONFIG__` in the browser, mirroring the backend behavior. It also serves as a prefix for `VITE_API_ENDPOINTS`, ensuring correct access to backend APIs. The matching `webuiPrefix` is derived as `${VITE_DEV_API_PREFIX}/webui/` automatically — you don't need a separate variable for it.
+
+Three scenarios cover everything you'll hit:
+
+### Scenario 1 — single-instance dev (no prefix, no proxy)
+
+The default. Don't set anything beyond the existing `.env.development`.
+
+```
+Browser ──► localhost:5173 (Vite) ──► localhost:9621 (backend, no prefix)
+```
+
+```bash
+# lightrag_webui/.env.development (already in repo as sample)
+VITE_BACKEND_URL=http://localhost:9621
+VITE_API_PROXY=true
+VITE_API_ENDPOINTS=/api,/documents,/graphs,/graph,/health,/query,/docs,/redoc,/openapi.json,/login,/auth-status,/static
+# VITE_DEV_API_PREFIX=          ← leave empty
+```
+
+Run:
+```bash
+lightrag-server                  # in one terminal, no LIGHTRAG_API_PREFIX
+cd lightrag_webui && bun run dev # in another; open http://localhost:5173/
+```
+
+### Scenario 2 — simulate a site prefix
+
+You want the SPA to run under `/site01` (or whatever production prefix). Set `VITE_DEV_API_PREFIX=/site01`. Vite injects the matching `window.__LIGHTRAG_CONFIG__` and registers prefixed proxy keys; SPA requests like `fetch("/site01/documents/foo")` are forwarded verbatim to whatever `VITE_BACKEND_URL` points at. The upstream — local backend or production nginx — is responsible for understanding the prefix.
+
+```
+Browser ──► localhost:5173 (Vite + HMR)
+                │
+                │  Vite proxy forwards /site01/* verbatim, no rewrite
+                ▼
+            VITE_BACKEND_URL  ──►  upstream that knows /site01
+```
+
+`.env.local` (gitignored — your personal dev config):
+```bash
+VITE_BACKEND_URL=…                             # see "Where to point VITE_BACKEND_URL" below
+VITE_API_PROXY=true
+VITE_API_ENDPOINTS=/api,/documents,/graphs,/graph,/health,/query,/docs,/redoc,/openapi.json,/login,/auth-status,/static
+VITE_DEV_API_PREFIX=/site01
+```
+
+Run `bun run dev` and open **`http://localhost:5173/`**. HMR is purely local — the browser only talks to `localhost:5173` for SPA assets, no WebSocket-upgrade config needed on any upstream.
+
+#### Where to point `VITE_BACKEND_URL`
+
+Two options, picked by where the prefix-aware upstream lives. The Vite-side configuration is identical; only this one variable changes.
+
+**A. Local backend with `LIGHTRAG_API_PREFIX=/site01`** (no nginx anywhere) — the simplest setup, two processes on your laptop. Vite's proxy itself plays the role of the reverse proxy.
+
+```bash
+VITE_BACKEND_URL=http://localhost:9621
+```
+```bash
+# Terminal 1
+LIGHTRAG_API_PREFIX=/site01 lightrag-server
+# Terminal 2
+cd lightrag_webui && bun run dev
+```
+
+The backend's FastAPI `root_path=/site01` accepts the prefixed form natively (Starlette's `get_route_path()` strips `root_path` from the request path before matching), so no extra rewriting is needed on either side.
+
+**B. Real (remote) backend reached through its production nginx** — useful when the actual backend has data / configs that are painful to reproduce locally. nginx already strips `/site01/` before forwarding to the backend; the dev frontend benefits without changing anything in production.
+
+```bash
+VITE_BACKEND_URL=https://prod.example.com      # or http://10.0.0.5 — the nginx URL
+```
+
+The production nginx and backend stay exactly as they are. The flow becomes:
+
+```
+SPA fetch /site01/documents/foo
+  → Vite forwards to https://prod.example.com/site01/documents/foo
+  → nginx matches /site01/, strips it, forwards /documents/foo to backend
+  → backend serves it
+```
+
+#### Why `VITE_BACKEND_URL` does **not** include `/site01`
+
+Vite forwards the request path **verbatim** (no rewrite). The browser already emits `/site01/documents/foo`, so the URL Vite sends upstream is `${VITE_BACKEND_URL}/site01/documents/foo`. If you set `VITE_BACKEND_URL=https://prod.example.com/site01` you would get `https://prod.example.com/site01/site01/documents/foo` — a duplicated prefix that both nginx and the backend reject. Always point `VITE_BACKEND_URL` at the upstream **root**.
+
+#### Common pitfalls (mostly relevant to option B)
+
+- **HTTPS upstream + self-signed cert**: Vite's proxy rejects by default. Set `proxy: { ..., secure: false }` in `vite.config.ts` to skip cert validation when targeting a staging proxy with a non-public cert.
+- **Auth required**: if the upstream requires `LIGHTRAG_API_KEY`, log in via the dev SPA exactly as you would in prod — the auth token flows through the proxy unchanged.
+- **CORS errors**: shouldn't happen because the browser sees same-origin requests to `localhost:5173`. If they appear, check that `changeOrigin: true` is in effect (it is, by default in `vite.config.ts`).
+
+### Quick decision matrix
+
+| Scenario | `VITE_BACKEND_URL` | `VITE_DEV_API_PREFIX` | Upstream the dev proxy talks to | Open in browser |
+| --- | --- | --- | --- | --- |
+| 1. Default single-instance dev | `http://localhost:9621` | unset | local backend, no prefix | `http://localhost:5173/` |
+| 2A. Simulate a prefix locally (no nginx) | `http://localhost:9621` | `/site01` | local backend with `LIGHTRAG_API_PREFIX=/site01` | `http://localhost:5173/` |
+| 2B. Hit a real backend through its production nginx | `https://prod.example.com` | `/site01` | remote nginx that already strips `/site01/` | `http://localhost:5173/` |
+
+Rows 2A and 2B share **everything except `VITE_BACKEND_URL`** — the choice is purely "is the prefix-aware upstream on my laptop or in production?".
+
+**The "Open in browser" column is always `http://localhost:5173/` — that is the entry point in every dev scenario.** What changes between rows is where the API traffic ultimately lands; the SPA itself is always served from the dev server's root.
+
+---
+
+## Troubleshooting
+
+### Asset URLs 404 when accessing the WebUI
+
+The base URL must end with `/`. Accessing `/site01/webui` (no trailing slash) makes the browser resolve `./assets/foo.js` against `/site01/`, which 404s. The server already redirects the no-slash form to the
+slash form; verify the redirect is reaching nginx (check `X-Forwarded-Prefix` and that nginx uses `proxy_pass http://…/` with the trailing slash).
+
+### `apiPrefix` is empty in `window.__LIGHTRAG_CONFIG__` after deploy
+
+View the page source. If you see the literal placeholder `<!-- __LIGHTRAG_RUNTIME_CONFIG__ -->` instead of an injected `<script>` tag, the request did not go through `SmartStaticFiles` — double-check that `lightrag/api/webui/index.html` exists in the running container and that the WebUI mount succeeded (the server logs `WebUI assets mounted at <path>` at startup).
+
+### `bun run dev` proxy returns 404 with `VITE_DEV_API_PREFIX` set
+
+Confirm the backend is also running with the matching `LIGHTRAG_API_PREFIX`. The dev proxy forwards prefixed paths verbatim; if the backend has no prefix configured, it does not register routes under that path.
+
+### I want to disable the WebUI entirely
+
+Don't build the frontend — `lightrag/api/webui/index.html` will not exist and the server will skip the WebUI mount, redirecting `/` and the WebUI path to `/docs` instead. The runtime-config injection is purely opt-in via the existence of the build artifact.

docs/OfflineDeployment.md

@@ -0,0 +1,316 @@
+# LightRAG Offline Deployment Guide
+
+This guide provides comprehensive instructions for deploying LightRAG in offline environments where internet access is limited or unavailable.
+
+If you deploy LightRAG using Docker, there is no need to refer to this document, as the LightRAG Docker image is pre-configured for offline operation.
+
+> Software packages requiring `transformers`, `torch`, or `cuda` will not be included in the offline dependency group. Consequently, document extraction tools such as Docling, as well as local LLM models like Hugging Face and LMDeploy, are outside the scope of offline installation support. These high-compute-resource-demanding services should not be integrated into LightRAG. Docling will be decoupled and deployed as a standalone service.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Quick Start](#quick-start)
+- [Layered Dependencies](#layered-dependencies)
+- [Tiktoken Cache Management](#tiktoken-cache-management)
+- [Complete Offline Deployment Workflow](#complete-offline-deployment-workflow)
+- [Troubleshooting](#troubleshooting)
+
+## Overview
+
+LightRAG uses dynamic package installation (`pipmaster`) for optional features based on file types and configurations. In offline environments, these dynamic installations will fail. This guide shows you how to pre-install all necessary dependencies and cache files.
+
+### What Gets Dynamically Installed?
+
+LightRAG dynamically installs packages for:
+
+- **Storage Backends**: `redis`, `neo4j`, `pymilvus`, `pymongo`, `asyncpg`, `qdrant-client`
+- **LLM Providers**: `openai`, `anthropic`, `ollama`, `zhipuai`, `aioboto3`, `voyageai`, `llama-index`, `lmdeploy`, `transformers`, `torch`
+- **Tiktoken Models**: BPE encoding models downloaded from OpenAI CDN
+
+**Note**: Document processing dependencies (`pypdf`, `python-docx`, `python-pptx`, `openpyxl`) are now pre-installed with the `api` extras group and no longer require dynamic installation.
+
+## Quick Start
+
+### Option 1: Using pip with Offline Extras
+
+```bash
+# Online environment: Install all offline dependencies
+pip install lightrag-hku[offline]
+
+# Download tiktoken cache
+lightrag-download-cache
+
+# Create offline package
+pip download lightrag-hku[offline] -d ./offline-packages
+tar -czf lightrag-offline.tar.gz ./offline-packages ~/.tiktoken_cache
+
+# Transfer to offline server
+scp lightrag-offline.tar.gz user@offline-server:/path/to/
+
+# Offline environment: Install
+tar -xzf lightrag-offline.tar.gz
+pip install --no-index --find-links=./offline-packages lightrag-hku[offline]
+export TIKTOKEN_CACHE_DIR=~/.tiktoken_cache
+```
+
+### Option 2: Using Requirements Files
+
+```bash
+# Online environment: Download packages
+pip download -r requirements-offline.txt -d ./packages
+
+# Transfer to offline server
+tar -czf packages.tar.gz ./packages
+scp packages.tar.gz user@offline-server:/path/to/
+
+# Offline environment: Install
+tar -xzf packages.tar.gz
+pip install --no-index --find-links=./packages -r requirements-offline.txt
+```
+
+## Layered Dependencies
+
+LightRAG provides flexible dependency groups for different use cases:
+
+### Available Dependency Groups
+
+| Group | Description | Use Case |
+| ----- | ----------- | -------- |
+| `api` | API server + document processing | FastAPI server with PDF, DOCX, PPTX, XLSX support |
+| `offline-storage` | Storage backends | Redis, Neo4j, MongoDB, PostgreSQL, etc. |
+| `offline-llm` | LLM providers | OpenAI, Anthropic, Ollama, etc. |
+| `offline` | Complete offline package | API + Storage + LLM (all features) |
+
+**Note**: Document processing (PDF, DOCX, PPTX, XLSX) is included in the `api` extras group. The previous `offline-docs` group has been merged into `api` for better integration.
+
+> Software packages requiring `transformers`, `torch`, or `cuda` will not be included in the offline dependency group.
+
+### Installation Examples
+
+```bash
+# Install API with document processing
+pip install lightrag-hku[api]
+
+# Install API and storage backends
+pip install lightrag-hku[api,offline-storage]
+
+# Install all offline dependencies (recommended for offline deployment)
+pip install lightrag-hku[offline]
+```
+
+### Using Individual Requirements Files
+
+```bash
+# Storage backends only
+pip install -r requirements-offline-storage.txt
+
+# LLM providers only
+pip install -r requirements-offline-llm.txt
+
+# All offline dependencies
+pip install -r requirements-offline.txt
+```
+
+## Tiktoken Cache Management
+
+Tiktoken downloads BPE encoding models on first use. In offline environments, you must pre-download these models.
+
+### Using the CLI Command
+
+After installing LightRAG, use the built-in command:
+
+```bash
+# Download to default location (see output for exact path)
+lightrag-download-cache
+
+# Download to specific directory
+lightrag-download-cache --cache-dir ./tiktoken_cache
+
+# Download specific models only
+lightrag-download-cache --models gpt-4o-mini gpt-4
+```
+
+### Default Models Downloaded
+
+- `gpt-4o-mini` (LightRAG default)
+- `gpt-4o`
+- `gpt-4`
+- `gpt-3.5-turbo`
+- `text-embedding-ada-002`
+- `text-embedding-3-small`
+- `text-embedding-3-large`
+
+### Setting Cache Location in Offline Environment
+
+```bash
+# Option 1: Environment variable (temporary)
+export TIKTOKEN_CACHE_DIR=/path/to/tiktoken_cache
+
+# Option 2: Add to ~/.bashrc or ~/.zshrc (persistent)
+echo 'export TIKTOKEN_CACHE_DIR=~/.tiktoken_cache' >> ~/.bashrc
+source ~/.bashrc
+
+# Option 3: Copy to default location
+cp -r /path/to/tiktoken_cache ~/.tiktoken_cache/
+```
+
+## Complete Offline Deployment Workflow
+
+### Step 1: Prepare in Online Environment
+
+```bash
+# 1. Install LightRAG with offline dependencies
+pip install lightrag-hku[offline]
+
+# 2. Download tiktoken cache
+lightrag-download-cache --cache-dir ./offline_cache/tiktoken
+
+# 3. Download all Python packages
+pip download lightrag-hku[offline] -d ./offline_cache/packages
+
+# 4. Create archive for transfer
+tar -czf lightrag-offline-complete.tar.gz ./offline_cache
+
+# 5. Verify contents
+tar -tzf lightrag-offline-complete.tar.gz | head -20
+```
+
+### Step 2: Transfer to Offline Environment
+
+```bash
+# Using scp
+scp lightrag-offline-complete.tar.gz user@offline-server:/tmp/
+
+# Or using USB/physical media
+# Copy lightrag-offline-complete.tar.gz to USB drive
+```
+
+### Step 3: Install in Offline Environment
+
+```bash
+# 1. Extract archive
+cd /tmp
+tar -xzf lightrag-offline-complete.tar.gz
+
+# 2. Install Python packages
+pip install --no-index \
+    --find-links=/tmp/offline_cache/packages \
+    lightrag-hku[offline]
+
+# 3. Set up tiktoken cache
+mkdir -p ~/.tiktoken_cache
+cp -r /tmp/offline_cache/tiktoken/* ~/.tiktoken_cache/
+export TIKTOKEN_CACHE_DIR=~/.tiktoken_cache
+
+# 4. Add to shell profile for persistence
+echo 'export TIKTOKEN_CACHE_DIR=~/.tiktoken_cache' >> ~/.bashrc
+```
+
+### Step 4: Verify Installation
+
+```bash
+# Test Python import
+python -c "from lightrag import LightRAG; print('✓ LightRAG imported')"
+
+# Test tiktoken
+python -c "from lightrag.utils import TiktokenTokenizer; t = TiktokenTokenizer(); print('✓ Tiktoken working')"
+
+# Test optional dependencies (if installed)
+python -c "import redis; print('✓ Redis available')"
+```
+
+## Troubleshooting
+
+### Issue: Tiktoken fails with network error
+
+**Problem**: `Unable to load tokenizer for model gpt-4o-mini`
+
+**Solution**:
+```bash
+# Ensure TIKTOKEN_CACHE_DIR is set
+echo $TIKTOKEN_CACHE_DIR
+
+# Verify cache files exist
+ls -la ~/.tiktoken_cache/
+
+# If empty, you need to download cache in online environment first
+```
+
+### Issue: Dynamic package installation fails
+
+**Problem**: `Error installing package xxx`
+
+**Solution**:
+```bash
+# Pre-install the specific package you need
+# For API with document processing:
+pip install lightrag-hku[api]
+
+# For storage backends:
+pip install lightrag-hku[offline-storage]
+
+# For LLM providers:
+pip install lightrag-hku[offline-llm]
+```
+
+### Issue: Missing dependencies at runtime
+
+**Problem**: `ModuleNotFoundError: No module named 'xxx'`
+
+**Solution**:
+```bash
+# Check what you have installed
+pip list | grep -i xxx
+
+# Install missing component
+pip install lightrag-hku[offline]  # Install all offline deps
+```
+
+### Issue: Permission denied on tiktoken cache
+
+**Problem**: `PermissionError: [Errno 13] Permission denied`
+
+**Solution**:
+```bash
+# Ensure cache directory has correct permissions
+chmod 755 ~/.tiktoken_cache
+chmod 644 ~/.tiktoken_cache/*
+
+# Or use a user-writable directory
+export TIKTOKEN_CACHE_DIR=~/my_tiktoken_cache
+mkdir -p ~/my_tiktoken_cache
+```
+
+## Best Practices
+
+1. **Test in Online Environment First**: Always test your complete setup in an online environment before going offline.
+
+2. **Keep Cache Updated**: Periodically update your offline cache when new models are released.
+
+3. **Document Your Setup**: Keep notes on which optional dependencies you actually need.
+
+4. **Version Pinning**: Consider pinning specific versions in production:
+   ```bash
+   pip freeze > requirements-production.txt
+   ```
+
+5. **Minimal Installation**: Only install what you need:
+   ```bash
+   # If you only need API with document processing
+   pip install lightrag-hku[api]
+   # Then manually add specific LLM: pip install openai
+   ```
+
+## Additional Resources
+
+- [LightRAG GitHub Repository](https://github.com/HKUDS/LightRAG)
+- [Docker Deployment Guide](./DockerDeployment.md)
+- [API Server Documentation](./LightRAG-API-Server.md)
+
+## Support
+
+If you encounter issues not covered in this guide:
+
+1. Check the [GitHub Issues](https://github.com/HKUDS/LightRAG/issues)
+2. Review the [project documentation](../README.md)
+3. Create a new issue with your offline deployment details

docs/ParagraphSemanticChunking-zh.md

@@ -0,0 +1,523 @@
+# Paragraph Semantic 分块策略
+
+## 1. 适用场景与策略选择
+
+### 1.1 P 策略要解决什么问题
+
+Paragraph Semantic Chunking（下文简称 **P 策略**）面向 DOCX、PDF 等具有清晰章节结构的文档。其核心目标是：**让分块边界尽可能对齐文档原生的语义边界**（标题、段落、表格行），而不是仅由 token 长度计数决定切点。
+
+P 策略主要解决以下四类问题：
+
+1. **表格语境断裂**：大表被拆分后，首尾切片容易脱离前置说明、后置解释或中间桥接文字，召回时无法独立理解。
+2. **层级信息利用不足**：仅看相邻段落的方法无法利用父标题路径、同级条款之间的关系。
+3. **细碎章节尺寸失衡**：规章、标准、合同等文档常包含大量 100~300 token 的细碎条款，若不合并则块过短、语义稀薄；若仅按相邻长度合并又会跨主题污染。
+4. **长块二次拆分破坏结构**：章节过长时，常规字符切分会忽略表格行边界和标题层级。
+
+P 策略对**任何能生成 `.blocks.jsonl` sidecar 的 parser**（`native` / `mineru` / `docling`）产出的结构化产物有效——三者经统一的 `write_sidecar()` 落盘相同结构的 `.blocks.jsonl`（含 `heading` / `level` / `parent_headings`）。仅 `legacy` 引擎不产出 sidecar；无 sidecar 的输入（legacy 路径或解析失败）会自动降级为 R 策略（见 §6）。
+
+### 1.2 P / R / V 三种策略对比
+
+| 维度 | R 策略（Recursive） | V 策略（SemanticVector） | P 策略（ParagraphSemantic） |
+|---|---|---|---|
+| 切分依据 | 字符分隔符级联（段落 → 换行 → 中文标点 → 空格 → 字符）+ token 预算 | 句子级 embedding 距离阈值（百分位 / 标准差 / 四分位距 / 梯度）寻找语义断层 | 标题 outline level 与 `parent_headings` + 表格行边界 + 锚点 + 层级感知合并 |
+| 块大小控制 | `chunk_token_size` 硬上限 | `chunk_token_size` 仅为 advisory ceiling，超限时通过 R 二次切分 | `target_max` 硬上限 + `target_ideal` 软目标 + 表格阈值 + 尾部吸收阈值多重协同 |
+| 表格处理 | 不感知表格，可能在表格中间切断 | 不感知表格 | 表格小于 `table_max` 保持完整；大表按 JSON 行数组 / HTML `<tr>` 行边界切片，并重新包裹为合法 `<table>` |
+| 表格上下文 | 依赖窗口偶然覆盖 | 依赖 embedding 距离 | 首切片粘连前置说明、末切片粘连后置解释、连续大表桥接文字双向重叠 |
+| 块间重叠 | 全局 `chunk_overlap_token_size` | 不会出现重叠 | 章节边界不会重叠；同章节长正文 fallback 到 R 时按 `CHUNK_P_OVERLAP_SIZE` 重叠；连续大表桥接文字可同时进入前后两个表格块 |
+| heading 元数据 | 通常无 | 通常无 | 继承或提升 heading；拆分后追加 `[part n]` 后缀；保留 `parent_headings` 和 `level` |
+| 嵌入计算开销 | 无 | 高（需对每个句子计算 embedding） | 无 |
+| 依赖输入 | 任意文本 | 任意文本 + Embedding 模型 | 必须有 `.blocks.jsonl` sidecar（`native` / `mineru` / `docling` 任一引擎产出），否则降级为 R |
+
+### 1.3 怎么选
+
+| 场景 | 推荐 | 理由 |
+|---|---|---|
+| 章节层级清晰（内容解析引擎需要能够生成Sidecar文件） | **P** | 充分利用标题层级与表格行边界，块边界最贴合语义；避免跨主题污染 |
+| 文档以散文 / 评论 / 长篇正文为主，没有明确章节结构 | **V** | 按语义相似度切分能在话题切换点形成自然边界，比字符切分更稳定 |
+| 输入是纯文本、Markdown、代码、日志，或追求最低算力开销 | **R** | 无嵌入开销，分隔符级联对中英文混合文本足够稳定 |
+| 通用配置（不确定文件类型） | **R** | P 在无 sidecar 时自动降级到 R；V 在无 Embedding 模型时也降级到 R |
+| 标题样式混乱、正文中大量伪标题的文档 | **R** 或 **V** | P 依赖 parser 正确识别标题，标题错乱会导致基础块边界偏移 |
+| 单行超大表格或不可解析表格 | 任意 | 三种策略最终都会走字符级 fallback；P 仍保留表格上下文粘连优势 |
+
+## 2. 设计目标与核心不变量
+
+P 策略的全部规则都服务于一个目标：**让块边界对齐文档原生语义边界，并让每个块在召回时能被独立理解**。它把这个目标拆成针对三类场景的具体规则（表格、长块、细碎章节），逐条在 §3 展开。无论规则如何组合，以下四条**重叠不变量**始终成立——它们界定了「哪里允许文字复制、哪里绝不允许」：
+
+1. **章节边界不会重叠**：不同 `.blocks.jsonl` 内容行之间的文本绝不会被复制到对方块里，避免“张冠李戴”。
+2. **章节内长正文可重叠**：同一个内容行内拆分的多个片段允许按 `chunk_overlap_token_size` 保留 R 风格 overlap，减少长正文中途切断。
+3. **表格之间桥接文字可双向重叠**：唯一的跨段落复制场景，专门服务连续大表的上下文保留。
+4. **表格行不互相重叠**：行级切片本身是非重叠的，与 R 的 overlap 概念不同。
+
+### 2.1 规则与效果速览
+
+下表把 §3 的每条规则映射到它达到的效果，以及实现它的内部阶段（阶段名同时是代码注释、日志关键字与排查的交叉引用标识，见 §7.6）：
+
+| 分块规则 | 达到的效果 | 实现阶段 | 详见 |
+|---|---|---|---|
+| 标题级基础块 | 块边界对齐文档原生结构，而非 token 计数 | HeadingBlocks | §3.1 |
+| 表格完整性 + 行边界切片 | 表格不被从中间截断，切片仍是合法 `<table>` | TableRowSplit | §3.2 |
+| 表格上下文粘连（角色 + 桥接双向重叠） | 表格前置说明、后置解释、桥接文字不脱离表格 | TableRowSplit / TableBridge | §3.3 |
+| 表头恢复（切分时补回中段/末段表头） | 被拆表的 `middle`/`last` 切片单独召回时不丢失列名，且不触发长度超限 | HeaderRecovery | §3.3.3 |
+| 锚点驱动长块再切分 | 超长章节按语义点切分，保留标题层级 | AnchorSplit | §3.4 |
+| 无正文标题粘连 | 父标题不与其子内容失散 | HeadingGlue | §3.5 |
+| 层级感知合并 | 细碎条款聚到理想大小，又不跨主题污染 | LevelMerge | §3.6 |
+| 重叠规则 | 召回上下文充分，但章节/表格边界不“张冠李戴” | 贯穿全程 | §3.7 |
+| 尺寸阈值协同 | 大多数块落在 `[target_ideal, target_max]` | 贯穿全程 | §3.8 |
+
+### 2.2 处理流水线总览
+
+上述规则串成一条以 `.blocks.jsonl` 为输入的流水线（**每个 `type == "content"` 行被视为一个标题级基础块**）：
+
+```text
+DOCX / PDF / PPTX / …
+  ↓  native(docx) / mineru / docling parser —— 按标题输出基础块，不做 token 拆分
+.blocks.jsonl + sidecar (.tables.json / .equations.json / .drawings.json / .blocks.assets/)
+  ↓  TableRowSplit：超大表格按行边界切片并赋予 first/middle/last 角色      → §3.2
+  ↓  HeaderRecovery：切分时把重复表头预扣预算后注入中段/末段切片      → §3.3.3
+  ↓  TableBridge：连续大表之间桥接文字双向重叠                        → §3.3
+  ↓  AnchorSplit：锚点驱动的长文本块再切分                              → §3.4
+  ↓  PartLabeling：[part n] 行级来源追溯编号（按原始内容行独立编号，故在跨行合并之前）
+  ↓  HeadingGlue：无正文标题块向前并入严格更深的子块               → §3.5
+  ↓  LevelMerge：层级感知的双相位合并                                  → §3.6
+最终 chunk 列表
+```
+
+## 3. 分块规则与效果
+
+### 3.1 标题级基础块——边界对齐文档原生语义 〔HeadingBlocks〕
+
+**规则**：每个 `.blocks.jsonl` 的 `type == "content"` 行就是一个基础块，即「一条标题下的正文作为一个块」。标题识别完全由 **parser** 完成，**P chunker 自身不扫描文档 body、也不判断标题样式**，更不在解析阶段做 token 阈值拆分。
+
+**达到的效果**：块的初始边界天然落在文档大纲结构上（标题切换处），而不是任意 token 位置；后续所有阶段都在这个语义对齐的基础上做加工。
+
+三个能产出 sidecar 的引擎殊途同归地按标题切出基础块，各自得到 `heading` / `level` / `parent_headings`：
+
+- **native（docx）**：读取 `styles.xml`，按 `<w:basedOn>` 建立样式继承链回溯有效 `<w:outlineLvl>`；遍历 `document.xml` 段落沿继承链解析大纲级别，原始 outline level 0~8 映射为内部 `level` 1~9；维护 `current_heading_stack`，遇新标题清理不浅于当前 level 的旧标题并计算 `parent_headings`。
+- **mineru**：按条目的 `text_level > 0` 或 `label` 为 `title` / `section_header` 检测标题，用 heading_stack 维护父链。
+- **docling**：`label="title"` → level 1，`label="section_header"` → `item.level + 1`（默认 level 2），同样维护父链。
+
+三者最终都产出统一的 `IRBlock`（携带 `heading` / `level` / `parent_headings`），并由 `write_sidecar()` 落为相同结构的 `.blocks.jsonl`；表格、公式、图形被提取为单行标签（`<table id="..." format="json">...</table>` 等）写入对应 sidecar。所有可识别标题均触发基础块边界，**不**执行 token 阈值拆分。
+
+P chunker 直接读取 `.blocks.jsonl`，每个 content 行作为后续 TableRowSplit/AnchorSplit 的独立处理单元——这也意味着 `[part n]` 编号按每个原始 content 行**独立重置**（见 §3.4 与 §4.4）。
+
+### 3.2 表格完整性与行边界切片——不从中间截断表格 〔TableRowSplit〕
+
+**规则**：token 数不超过 `table_max` 的表格**保持完整**；只有超过 `table_max` 的表格才切片，且**优先按行边界切**，只有收敛到单行仍无法在上限内表达时，整张表才退化为字符级切分。
+
+**达到的效果**：表格永远不会在「单元格中间」被截断；每个切片都重新包裹为合法的 `<table>` 标签，下游解析与 LLM 阅读都能把它当作表格理解，而非破碎的标记片段。
+
+#### 3.2.1 行边界优先切片
+
+- `format="json"`：按 JSON 顶层行数组切片。
+- `format="html"`：按 `<tr>...</tr>` 行切片。
+- 未显式标注但内容可嗅探为 JSON / HTML 的表格同样按上述规则处理。
+
+切片前预扣 `<table {attrs}></table>` 外壳 token 开销，使重新包裹后的切片尽量不超过 `table_max`。每个切片重新包裹为合法的 `<table>` 标签，便于下游解析。
+
+#### 3.2.2 行级递归二次切片
+
+若某个行子集重新包裹后仍超过 `table_max`，则在该行子集内继续细分。**当切片收敛到单行、且该单行无法在 `target_max` 内与表头并存（单行内容本身超上限，或没超但加上表头后超上限）时，整张表退化为对原始 `<table>` 文本（其 body 天然含表头）的 R 递归字符切分，并打一条 `logger.warning` 警告**——表头内容随原表文本以纯文本形式保留，绝不被静默丢弃，也不产生「部分 `<table>` 切片 + 部分孤立字符片段」的混合输出。不需要注入表头、且单行本身装得进 `target_max` 的切片仍原样保留为合法 `<table>` 标记。该机制使可被行边界表达的表格内容尽量保留合法表格结构。
+
+#### 3.2.3 末片回吞
+
+若表格末片 token 数低于 `table_min_last`，且与前一切片合并后不超过 `table_max`，则将末片回吞至前一切片，减少无效短表格块。
+
+### 3.3 表格上下文粘连——前后说明、桥接与表头不失散 〔TableRowSplit / TableBridge / HeaderRecovery〕
+
+**规则**：被切片的表格按「首/中/末」角色与周围段落做差异化粘连；连续两张大表之间的短桥接文字按预算**双向**分配到两侧表格块；丢失表头的中段/末段切片在**切分时**就把该表的重复表头拼回自身的 `<table>`（表头 token 已在切分前预扣进每片上限）。
+
+**达到的效果**：表格的**前置说明**进入首切片块、**后置解释**进入末切片块、**桥接文字**同时作为左表后文与右表前文——任一表格切片被召回时都带着足以独立理解的上下文，不会出现「表格在这、解释在另一块」的断裂。被拆表的中段/末段切片即使脱离了承载表头的首切片，也会把表头行重新拼回自身的 `<table>` 开头，使其单独召回时仍能理解每列含义。
+
+#### 3.3.1 表格切片角色与物理粘连
+
+每个表格切片被赋予内部字段 `table_chunk_role`，并按角色决定与周围段落的粘连方式：
+
+| 角色 | 含义 | 粘连策略 |
+|---|---|---|
+| `first` | 原始表格的首切片 | 追加到当前累积块尾部，使表格**前置说明**与首切片进入同一块 |
+| `middle` | 原始表格的中间切片 | 独立输出，避免与无关正文合并 |
+| `last` | 原始表格的末切片 | 作为新累积块起点，使**后置解释**自动追加到末切片之后 |
+| `none` | 非表格切片或未拆分的完整表格 | 按普通文本块处理 |
+
+`table_chunk_role` 是内部字段，最终输出不会保留，**但在 LevelMerge 中继续作为合并约束使用**（见 §3.6.1）。
+
+#### 3.3.2 连续大表桥接文字双向重叠 〔TableBridge〕
+
+当同一原始内容行中出现「大表 A、短桥接文字、大表 B」的模式，且两张表均被拆分时，桥接文字按上下文预算进行双向分配：
+
+1. 将桥接文字按 token 编码。
+2. 计算左侧预算 `prev_budget = min(chunk_overlap_token_size, target_max - 左侧末切片当前 token 数)`。
+3. 计算右侧预算 `next_budget = min(chunk_overlap_token_size, target_max - 右侧首切片当前 token 数)`。
+4. **若桥接文字长度同时不超过两侧预算**：左右两个表格边界块都包含**完整桥接文字**。
+5. **若桥接文字较长**：前缀进入左侧末切片块，后缀进入右侧首切片块；超出两侧预算的中间段独立成为普通文本块。该中间段块**与左右两侧各保留 `chunk_overlap_token_size` 的 R 风格 overlap**：向左回吞已进入左表块的前缀尾部、向右多含已进入右表块的后缀头部。由于每侧前缀/后缀长度本身就 ≤ overlap 预算，overlap 区间会覆盖整段前缀与后缀，**结果中间段块实际承载完整桥接文字**（桥接文字因此从不被切散，只是其首尾**额外**复制进相邻表格块）。overlap 索引始终夹在桥接文字 token 内，**绝不会把 `<table>` 内容拷进中间段块**。
+
+单侧预算还会被限制到不超过 `chunk_token_size / 2`，避免桥接文字主导整个块。
+
+这与普通相邻 chunk overlap 的差异：
+
+- 普通 overlap 按前后顺序复制字符或 token，与边界类型无关。
+- TableBridge 机制以表格切片角色为触发条件，把桥接文字同时作为左表后文上下文和右表前文上下文，避免桥接说明只归属一侧表格或被单独切散后难以召回。
+
+#### 3.3.3 中段/末段切片表头恢复 〔HeaderRecovery〕
+
+大表按行边界切片后，表头行只保留在**首切片**内；`middle` / `last` 切片因此丢失列名，单独召回时无法判断每列含义。为此在 **TableRowSplit 切分的同时**，把表头行直接拼回非首切片自身的 `<table>`，使每个切片重新成为带表头的完整表格。
+
+1. **表头来源**：解析期已把每张表的「跨页重复表头」写入同目录的 `.tables.json`（条目字段 `table_header`；**只有真正带重复表头的表才有该字段**）。**该字段按表格自身格式原生存储，使合并单元格语义全程存活**：`format="json"` 表存为 JSON 二维数组字符串（如 `[["H1","H2"]]`），`format="html"` 表存为原始 `<thead>…</thead>` 片段（保留 `rowspan` / `colspan`）。P 按待拆 `<table>` 标签保留的 `id` 关联回对应表条目，取其 `table_header`。
+2. **预扣预算、切分时注入**：表头的 token 数在切分**之前**就从每片 body 上限中预扣（与 `<table {attrs}></table>` 包裹开销一并扣除）。`_split_table_text` 据此切分，再把表头拼回每个非首切片——`format="json"` 切片把表头行 prepend 到行数组，`format="html"` 切片把存储的原始 `<thead>` 片段 **verbatim 拼回 body 开头**（保留 `rowspan` / `colspan` 合并单元格语义，不再展开为无 span 的网格）；若 HTML 切片已自带 `<thead>`（切点落在多行表头内部）则跳过，避免重复。切片原有 `attrs`（含首位的 `id`）保持不变。由于已预扣，**切片含表头后仍 ≤ `target_max`**，硬上限由所有下游阶段自然保证，不存在事后回填导致的超限。首切片自带真实表头行，不重复注入。若某切片收敛到单行后已无法在 `target_max` 内同时容纳行内容与表头（见 §3.2.2），则**整张表退化为 R 递归字符切分（含表头）并打 warning**，绝不保留无表头的孤立切片。
+3. **绝不臆造表头**——以下情形均不注入：源表在 `.tables.json` 中没有 `table_header` 字段（无重复表头）、`.tables.json` 缺失/不可读、切片已退化为字符级非 `<table>` 片段（无 `id` 可关联），或表格未发生真正的多片切分。
+4. **格式一致性硬校验（损坏即报错）**：注入前先判定 `table_header` 的格式（JSON 二维数组 vs `<thead>` 片段）并与待拆表格自身的 `format` 比对。两者明确冲突（如 HTML 表却拿到 JSON 数组表头，反之亦然）意味着 sidecar 已损坏或张冠李戴，此时 **`_split_table_text` 直接 `raise ValueError` 中断该文档分块**，而非用错位表头产出畸形切片。这是刻意的「损坏即硬报错」语义，与第 3 点「表头缺失则静默跳过」相区分——缺失是可容忍的常态，格式冲突是数据损坏信号。
+
+> 因为表头在切分时即进入切片，被拆表的各切片在 LevelMerge 中被**完全冻结、互不重新合并**（见 §3.6.1）——否则把同一张表的两个切片重新合并会在表中重复一次表头。表头**进入 `content`**，计入该 chunk 的 token 数（表头通常很小）；不写入 `heading`。
+
+### 3.4 锚点驱动的长块再切分——按语义点切、保留标题 〔AnchorSplit〕
+
+**规则**：对 TableRowSplit 后仍超过 `target_max` 的内容块，优先在「短段落锚点」处均衡切分，被选中的锚点晋升为子块新标题；无合格锚点时按「表格优先 → 贪心打包 → 字符切分」三级降级。
+
+**达到的效果**：超长章节不是被硬切在任意 token 位置，而是切在短小的小标题/过渡句这类**自然语义点**上，子块继承可读的标题与父标题路径；同时保证算法**永不丢内容**，且尽量遵守用户配置的块大小上限。
+
+#### 3.4.1 短段落锚点
+
+把内容按段落恢复，选择满足以下条件的段落作为候选锚点：
+
+- 段落不是表格（不以 `<table` 开头）。
+- 段落文本长度不超过 `max_anchor_candidate_length`（100 字符）。
+- 段落不是该块的第一个段落（避免递归无法收敛）。
+
+#### 3.4.2 均衡选锚
+
+根据目标子块数量计算理想切分位置，从候选锚点中选择距离理想位置最近的锚点。被选中的锚点**晋升为后续子块的新 `heading`**，原 heading 写入该子块的 `parent_headings`。
+
+#### 3.4.3 无锚点降级
+
+若不存在合格锚点：
+
+1. **表格优先**：若块内仍存在超限表格，优先调用 TableRowSplit 的行边界切片。
+2. **贪心打包**：其余文本按段落贪心打包到接近 `target_max`。
+3. **递归字符切分**：单一过长普通文本段落降级到 R 策略（`chunking_by_recursive_character`），使用 `chunk_overlap_token_size` 保持相邻文本片段的连续性。
+
+无锚点 fallback 路径保证算法**不会丢弃内容**，并尽量遵守用户配置的块大小上限。
+
+### 3.5 无正文标题粘连——父标题不与子内容失散 〔HeadingGlue〕
+
+**规则**：当一个块是 heading-only（只有标题、没有自己的正文）且紧邻的下一块层级**严格更深**时，把它**向前并入**那个更深的子块，并保留较浅的**父标题**身份；其余情况原样留给 LevelMerge。
+
+**达到的效果**：像 `## 2.4`（无正文）这样的父标题，绝不会被单独切成孤块、再被 LevelMerge 向后吞进上一个同级块 `## 2.3` 而与它真正的子内容 `### 2.4.1` 失散——标题始终随其子内容走，标题路径层级无损。
+
+某些章节只有标题、没有自己的正文（heading-only），例如：
+
+```
+## 2.3   结构尺寸及重量 .....   （level 2，有正文）
+## 2.4   环境适应性指标          （level 2，heading-only，无正文）
+### 2.4.1   概述                 （level 3，有正文）
+```
+
+若直接进入 LevelMerge，`## 2.4` 会作为一个独立的同级小块，被 Phase A 同级合并或尾部整批吸收**向后吞进上一个同级块 `## 2.3` 的末端**，使这个父标题与它真正的子内容 `### 2.4.1` 失散。
+
+因此在 LevelMerge 之前增加一个前置步骤（`_glue_heading_only_blocks`）。当前块为 heading-only（`content` 仅由标题行构成，由 `^#{1,6} +` 判定）时，**仅向前粘连**：
+
+- **触发条件**：紧邻的下一块层级**严格更深**（`level` 更大），且其 `table_chunk_role` 为 `none` 或 `first`。`first` 即「被切大表的首切片」——子节正文若是超大表格，TableRowSplit 切片后其首个产出块的角色为 `first`；紧跟 heading-only 行的只可能是下一行的首个产出块，故角色必为 `none` 或 `first`（`middle`/`last` 只在同一行表格内部出现）。
+- **并入 `first` 切片时保留其角色**：把 `## 2.4` 并入 `first` 切片后，合并块**仍标记 `first`**（`## 2.4` 标题正是表格的前置上下文，本就该由 `first` 切片承载）。这样 LevelMerge 不会把它向后吸回 `## 2.3`（`first` 不可被向后吸收），表格边界保护得以维持；`none` 子块的行为与现状完全一致。
+- **动作**：向前并入该子块，保留**父标题**身份（`heading` / `level` / `parent_headings` 取自较浅父块）。即 `## 2.4` 与 `### 2.4.1` 绑成一块，标题路径仍以 `2.4` 为主——子块 2.4.1 的 `parent_headings` 本就含 2.4，层级信息无损。链式标题（`# 2` → `## 2.4` → `### 2.4.1`）沿链折叠、保留**最浅**身份，直到遇到首个含正文的子块。
+- **不做向后粘连**：当下一块**不更深**（更浅/同级标题，或已到末尾）时，该 heading-only 块原样留给 LevelMerge。**不会**把它向后并入更深的前块（如 `### 2.3.9`）——把更浅的 `## 2.4` 标题吞进更深的 L3 块会倒置层级（深吞浅）、压低标题层级。这类孤立标题直接交给 LevelMerge 正常处理。
+- **保住硬上限**：子块来自 AnchorSplit、本在 `target_max` 内，但前缀拼入父标题行后可能超限。由于下游无人会重切超限块（LevelMerge 只阻止其继续变大），绑定后超限的块在此重切：**先剥离开头的标题行**，正文按**完整 `target_max`** 切分（使后续不含前缀的正文片保持完整预算），再把标题前缀拼回**第一个正文片**。仅当第一个正文片大到放不下前缀时，才单独对它用缩减后的上限再切——因此大前缀不会把整个子节切得过碎。这样标题始终随真实正文，绝不会被单独切成一个 heading-only 孤块（否则 LevelMerge 又会把它向后吸走），且每个产出片段仍 ≤ `target_max`。（退化情形：当前缀本身已吃满上限——极长标题或极小 `chunk_token_size`——无法保持完整，则整块直接切分、对超长标题行做字符级切分;此时 cap 优先于保持标题完整。）
+- **不额外回填前块**：由于 `keep="left"` 保留父块的 `level`，绑定后的整体只是个普通小块（并非锁定独立）。是否并回前块 `2.3` 完全沿用 LevelMerge 既有规则——前块仍 < `target_ideal` 时走同级合并，或整体小于 `small_tail_threshold` 时走尾部吸收（即便前块已饱和也能被吸入），二者都以重测后的真实 token ≤ `target_max` 为界。本前置步骤只保证标题**不脱离其子内容**，并不把整体锁为独立块——因此在尺寸允许时让 `2.3 + 2.4 + 2.4.1` 同块正是期望的防过碎行为。
+
+> 边界歧义：正文行若真以 `#␠` 开头会被误判为标题行——这是 `lightrag/parser/_markdown.py` 已记录并接受的同一启发式歧义，实际语料中概率极低。
+
+### 3.6 层级感知合并——细碎条款聚到理想大小、不跨主题污染 〔LevelMerge〕
+
+**规则**：**自深层级向浅层级处理**，先合并同级小块（Phase A），再尾部整批吸收，最后允许浅层块吸收深层块（Phase B）；每次合并都要同时满足尺寸、表格角色、层级、父标题路径四类约束。
+
+**达到的效果**：大量 100~300 token 的细碎条款被合并到接近 `target_ideal` 的尺寸（块不再过短、语义稀薄），同时**绝不把分属不同主题/不同父章节的相邻小块揉在一起**——既治「块过碎」，又防「跨主题污染」。
+
+#### 3.6.1 合并约束（每次合并都要满足）
+
+1. **尺寸约束**：合并后的真实文本 token 数不超过 `target_max`；已达到 `target_ideal` 的块原则上不继续参与普通同级合并。
+2. **角色约束（切片冻结）**：被拆表的所有切片 `first` / `middle` / `last` 一律**锁定独立、不参与任何合并**（既不向后吸收、也不被前块吸收、也不进入尾部整批吸收）。原因：表头已在 TableRowSplit 切分时注入各切片，若把同一张表的两个切片重新合并会在表中重复一次表头（§3.3.3）。表格边界的前后说明已在切分阶段粘进首/末切片，冻结不影响上下文粘连，只放弃小的首/末切片块与无关邻块的事后整合。仅 `none`（普通块/未拆分的完整表）可参与合并。
+3. **层级约束**：同级合并在相同 `level` 之间发生；跨级吸收只允许浅层吸收深层，**禁止深层反向吸收浅层**。
+4. **父标题路径一致性约束**：避免跨主题污染的关键，按合并方向取严格语义——
+   - **同级合并（Phase A / 尾部吸收）**：两块 `parent_headings` 必须**完全相等**（真·兄弟）。仅 `level` 相同但父链不同（如 `2.4.1` 与 `2.5.1`）不允许合并。
+   - **跨级吸收（Phase B，浅吸深）**：深块必须是浅块的**后代**——浅块的完整标题路径（`parent_headings` + 自身 `heading`，已剥离 `[part n]`）是深块 `parent_headings` 的前缀。浅块吞并不同分支的深块被禁止。
+   - `parent_headings` 为空（preamble / 无层级输入）的块视为路径相容，放行（无层级可污染）。
+
+#### 3.6.2 Phase A：同级合并
+
+针对当前 level 的相邻块，当**当前块**低于 `target_ideal`、合并后真实 token ≤ `target_max`，且满足上述约束时，合并为一个块（被吸收的邻块不要求低于 `target_ideal`；反向合并时另要求前块也 < `target_ideal`）。
+
+表格切片角色的方向规则（被拆表切片全部冻结，仅 `none` 可合并）：
+
+| 块角色 | 可向后吸收下一块 | 可被前一块吸收 |
+|---|:-:|:-:|
+| `none` | 是 | 是 |
+| `first` | 否 | 否 |
+| `middle` | 否 | 否 |
+| `last` | 否 | 否 |
+
+#### 3.6.3 尾部整批吸收
+
+若一个**普通块（`none`）**且已达到 `target_ideal` 的块后面紧跟一串同级小块，且该串小块总 token 数低于 `small_tail_threshold`、合并后真实 token 数不超过 `target_max`，则**一次性吸收**该串小块。遇到**任何被拆表切片**（`first` / `middle` / `last`），或父标题路径发生分叉时停止；被拆表切片自身也不会发起尾部吸收。
+
+#### 3.6.4 Phase B：跨级吸收
+
+对于 Phase A 后仍未饱和的小块，尝试跨级合并，但仅允许浅层吸收深层：
+
+- 当前块比后一块更浅时，当前块可向后吸收后一块。
+- 当前块比前一块更深时，前一浅层块可吸收当前块。
+- 反方向合并被禁止。
+- 被拆表切片（`first` / `middle` / `last`）在跨级阶段同样冻结，不参与合并；仅 `none` 块参与跨级吸收。
+
+#### 3.6.5 合并后真实 token 复测
+
+由于合并时会插入换行连接符，逐块 token 数相加可能低估合并结果。**每次提交合并前，都要对拼接后的真实文本重新计算 token 数**，确认不超过 `target_max` 后再提交。
+
+合并后保留主块的 `heading`。如果多个 part 片段被合并，最终 heading 保留主块的 part 后缀，**不会**额外拼接多个 part 标签。
+
+### 3.7 重叠规则汇总——哪里重叠、哪里绝不
+
+**规则 + 效果**：P 策略对「文字复制（overlap）」有精确的边界划分，既保证召回上下文充分，又杜绝跨章节/跨表格的“张冠李戴”。把散落在各阶段的重叠行为集中如下：
+
+| 场景 | 是否重叠 | 预算 / 机制 | 服务的效果 |
+|---|---|---|---|
+| 不同 `.blocks.jsonl` 内容行（章节边界） | **绝不重叠** | —— | 章节边界清晰，不张冠李戴 |
+| 同一内容行内长正文 fallback 到 R | 可重叠 | `chunk_overlap_token_size` | 长正文中途切断处保持语义连续 |
+| 连续大表之间的桥接文字 | 双向重叠 | 两侧各 `min(overlap, …, target_max/2)` | 桥接说明同时作为左右两表的上下文 |
+| 桥接长文本的独立中间段块 | 与左右各重叠 | `chunk_overlap_token_size`（夹在桥接 token 内，绝不含 `<table>`） | 中间段与相邻表格块阅读连续 |
+| 表格行级切片之间 | **绝不重叠** | —— | 行切片非重叠，避免重复行 |
+
+### 3.8 尺寸阈值协同——大多数块落在 [ideal, max]
+
+**规则**：P 策略的阈值不是固定常量，而是按 `chunk_token_size`（记为 N）动态推导，多个阈值协同控制文本块与表格切片的大小。
+
+**达到的效果**：理想分布下，大多数 chunk 落在 `[target_ideal, target_max]` 区间（N=2000 时约 1500~2000 token）；明显偏小的块通常只是锁定独立的 `middle` 表格切片或章节边界尾块。
+
+| 名称 | 计算式 | N = 2000 时取值 | 技术含义 |
+|---|---|---:|---|
+| `target_max` | N | 2000 | 文本块硬上限 |
+| `target_ideal` | 0.75 × N | 1500 | 文本块理想目标，达到此值后停止参与普通同级合并 |
+| `table_max` | 0.625 × N | 1250 | 表格触发切片阈值 |
+| `table_ideal` | 0.375 × N | 750 | 表格切片理想大小 |
+| `table_min_last` | 0.32 × `table_max` | 400 | 表格末片回吞阈值（小于此值且能合并则回吞至前一切片） |
+| `small_tail_threshold` | 0.125 × N | 250 | 尾部碎块吸收阈值 |
+| `max_anchor_candidate_length` | 固定 | 100 字符 | 长块拆分锚点候选段落长度上限 |
+
+比例约束关系：`table_max < target_ideal < target_max`、`table_ideal < table_max`。这些比例源自审计模式经验值（`大块 8000、小表 5000、理想表 3000、表格尾块 1600`），现按 `chunk_token_size` 等比缩放。
+
+## 4. 输入与输出
+
+### 4.1 输入
+
+`chunking_by_paragraph_semantic()` 接收以下输入：
+
+| 参数 | 来源 | 说明 |
+|---|---|---|
+| `content` | `full_docs[doc_id].content` | 拼接后的合并文本，用于 sidecar 缺失时降级 |
+| `blocks_path` | `full_docs[doc_id].lightrag_document_path` | `.blocks.jsonl` 路径，是 P 策略的主输入 |
+| `.tables.json`（隐式） | 由 `blocks_path` 推导（`<base>.blocks.jsonl` → `<base>.tables.json`） | HeaderRecovery（§3.3.3）的表头数据源；缺失时静默跳过表头注入 |
+| `chunk_token_size` | `chunk_options.chunk_token_size` / `CHUNK_P_SIZE` | 目标硬上限 N，默认 `2000` |
+| `chunk_overlap_token_size` | `CHUNK_P_OVERLAP_SIZE` / `chunk_overlap_token_size` | 同一内容行内长正文 fallback 与表格桥接预算的上限，默认 `100` |
+| `drop_references` | hint `drop_references`（别名 `drop_rf`）/ `CHUNK_P_DROP_REFERENCES` | 是否在分块前丢弃文末参考文献块，默认 `False`；**入队冻结进 `chunk_options`，并记录到 `doc_status.metadata['chunk_opts']`**（开启时记为 `drop_rf=True`） |
+| `references_tail_n` | `CHUNK_P_REFERENCES_TAIL_N` | 参考文献块只在文末最后 N 个内容块内才被丢弃（安全窗口），默认 `2`；**运行时实时读 env，不快照、不进 metadata** |
+| `references_headings` | `CHUNK_P_REFERENCES_HEADINGS`（竖线分隔） | 参考文献标题前缀，默认 `References\|Bibliography\|参考文献`；英文按单词边界、大小写不敏感匹配，`参考文献` 按前缀匹配；**运行时实时读 env，不快照、不进 metadata** |
+| `tokenizer` | LightRAG 已解析好的 tokenizer | 所有 token 计数与文本 overlap 截取的基准 |
+
+P 策略**不接收** `split_by_character` / `split_by_character_only`，因为正常路径由标题和段落结构驱动。
+
+**丢弃参考文献（`drop_references`）**：开启后，在 HeadingBlocks 之后、TableRowSplit/AnchorSplit/LevelMerge 之前，对从 `blocks.jsonl` 读出的有序内容块做过滤——**同时满足**「位于最后 `references_tail_n` 个块」且「`heading` 命中参考文献前缀」的块被丢弃。只有开关 `drop_references` 可经 per-file hint 设定并冻结进快照/metadata；`references_tail_n` / `references_headings` 是纯 env 调参，由 chunker 每次运行时实时读取当前环境变量——改 env 即可即时影响已入队文档的重跑。若丢弃后没有任何含内容的块剩余，则放弃丢弃并告警，避免产出空文档。
+
+### 4.2 `.blocks.jsonl` 约定
+
+P 策略只处理 `type == "content"` 行。每个内容行通常包含：
+
+- `content`：该标题下的正文文本，可能包含普通段落、`<table ... />` 标签、`<equation ... />` 公式、`<drawing ... />` 图形。
+- `heading`：当前标题。
+- `parent_headings`：父级标题链。
+- `level`：标题级别（1~9，对应原始 outline level 0~8）。
+- `positions`：原始段落定位（用于追溯）。
+- `blockid`：该内容行的稳定标识（可选）。存在时会被带入最终 chunk 的 `sidecar` 字段，供多模态管线与文档删除按源 block 回溯；缺失时（raw / legacy 输入）输出不含 `sidecar`。
+
+parser 保证「一条标题下的正文作为一个基础块」（native 经按标题的结构化切分，mineru / docling 经各自 IR builder），不在解析阶段做 token 阈值拆分。表格保持完整插入到 `content` 中。
+
+### 4.3 输出
+
+最终输出为有序 chunk 列表，每个元素：
+
+```python
+{
+    "tokens": int,                    # 真实 token 数（合并后会复测）
+    "content": str,                   # 块文本（可能包含 <table> 标签）
+    "chunk_order_index": int,         # 块顺序索引
+    "heading": {                      # 标题元数据（嵌套 dict，非扁平字段）
+        "level": int,                 # 标题层级
+        "heading": str,               # 拆分后追加 [part n] 后缀
+        "parent_headings": list[str], # 父级标题链，不追加后缀
+    },
+    # 可选：仅当输入 .blocks.jsonl 行带 blockid 时出现，
+    # 供多模态管线与文档删除按源 block 回溯。
+    "sidecar": {
+        "type": "block",
+        "id": str,                    # 主块 blockid（refs[0]）
+        "refs": [{"type": "block", "id": str}, ...],  # 去重后的全部源 blockid
+    },
+}
+```
+
+注意：`level` 与 `parent_headings` 现已收进 `heading` 嵌套 dict，顶层不再单独提供；`[part n]` 后缀落在 `heading["heading"]` 上。
+
+实现内部还会临时使用 `paragraphs`、`content`、`table_chunk_role`、`blockids` 等字段辅助拆分和合并，但**不会**以这些名字进入最终输出（`blockids` 经转换后体现为 `sidecar`）。
+
+### 4.4 `[part n]` 后缀规则
+
+- 同一个原始 `.blocks.jsonl` 内容行被拆成多个片段时，所有片段的 `heading` 字段追加 `[part 1]`、`[part 2]` …
+- 未发生拆分的内容行保持原 heading 不变。
+- `parent_headings` 不追加后缀。
+- 编号在每个原始内容行内**独立重置**（因 PartLabeling 在跨行合并之前编号，见 §2.2）。
+- 旧的 `[表格片段N]` 后缀已统一由 `[part n]` 替代。
+
+## 5. 配置项
+
+| 配置 | 默认 | 说明 |
+|---|---|---|
+| `CHUNK_P_SIZE` | `2000`（未设时使用 `DEFAULT_CHUNK_P_SIZE`，**不**沿用 `CHUNK_SIZE`） | P 专用 `chunk_token_size`；段落语义合并需要比全局默认更大的上限，因此独立默认而非回退到 `CHUNK_SIZE` |
+| `CHUNK_P_OVERLAP_SIZE` | 未设（沿用 `CHUNK_OVERLAP_SIZE`） | P 专用 overlap；只影响同一内容行内长正文 fallback 和表格桥接预算，**不**让表格行级切片互相重叠 |
+| `CHUNK_OVERLAP_SIZE` / `LightRAG(chunk_overlap_token_size=…)` | `100` | 未设 P 专用 overlap 时的全局兜底 |
+
+配置语法、优先级链、`addon_params["chunker"]` 运行时改值等详见 [FileProcessingConfiguration-zh.md](FileProcessingConfiguration-zh.md) §3。
+
+`P` 是与引擎正交的 chunking 选项（`后缀:引擎-选项`），可与任何产出 sidecar 的引擎组合。启用 P 的典型 `LIGHTRAG_PARSER` 写法：
+
+```bash
+# docx 用 native，pdf 用 mineru，其余支持格式用 docling，都启用 P；不支持的格式回退 legacy-R
+LIGHTRAG_PARSER=docx:native-teP,pdf:mineru-iteP,*:docling-iteP,*:legacy-R
+CHUNK_P_SIZE=2000
+CHUNK_P_OVERLAP_SIZE=100
+```
+
+（选项位 `i`/`t`/`e` 分别为图/表/公式分析，`P` 为 chunking 策略，可按需组合。）或在单文件覆盖：
+
+```text
+my-proposal.[native-P].docx
+paper.[mineru-P].pdf
+```
+
+## 6. 降级保护——永不丢内容
+
+**规则 + 效果**：P 策略有多层降级保护，任何结构化能力失效时都退到字符级切分，**保证文档仍产生检索块，不因结构化 sidecar 缺失而被静默丢弃**。
+
+| 触发条件 | 降级行为 |
+|---|---|
+| `blocks_path` 缺失、不可读、无有效 content 行 | 整体降级到 `chunking_by_recursive_character()`，传入解析出的 `chunk_overlap_token_size` |
+| TableRowSplit 中表格无法识别 JSON / HTML 结构 | 该表格调用 R 策略字符切分 |
+| TableRowSplit 中单行无法在 `target_max` 内与表头并存（单行内容超上限，或加表头后超上限） | **整张表（含表头）退化为 R 策略字符切分，并打 `logger.warning`**；表头内容随原表文本以纯文本保留 |
+| AnchorSplit 中长块没有合格短段落锚点 | 表格优先 → 贪心打包 → 单段落超长再降级 R 字符切分 |
+| HeaderRecovery 时 `.tables.json` 缺失/不可读、源表无 `table_header` | 跳过表头注入（该表本就无重复表头，不影响其余分块） |
+
+**重要**：整体 fallback 后不再具备标题层级、表格角色和桥接文字双向重叠能力；但能保证文档仍产生检索块。
+
+## 7. 效果检验与调试
+
+### 7.1 检查 sidecar 是否生成
+
+确认 parser 是否成功产生 `.blocks.jsonl`：
+
+```bash
+ls -l INPUT/__parsed__/<doc>.<ext>.parsed/<doc>.blocks.jsonl
+```
+
+若文件不存在或为空，P 策略会整体降级为 R，不会获得 P 的任何收益。常见原因：
+
+- 未给该格式配置能产出 sidecar 的引擎（如 `LIGHTRAG_PARSER=docx:native-...` / `pdf:mineru-...` / `*:docling-...`），实际走了 `legacy` 路径。
+- 解析失败（看 `pipeline_status` 错误条目）。
+- 该格式不被所选引擎支持（如 native 仅支持 docx；换用 mineru / docling 覆盖更多格式）。
+
+### 7.2 检查 blocks.jsonl 内容
+
+每行一个 JSON，过滤 `type == "content"` 后查看 heading / level / parent_headings 是否符合预期：
+
+```bash
+jq -c 'select(.type=="content") | {level, heading, parent_headings}' \
+   INPUT/__parsed__/<doc>.<ext>.parsed/<doc>.blocks.jsonl | head
+```
+
+若 heading 大量为空或 level 异常，说明 parser 没正确识别标题 —— 此时 P 策略的层级合并和锚点提升都会失效。
+
+### 7.3 检查最终 chunks 是否达到预期效果
+
+查看 `text_chunks` 存储中的 chunk 元数据：
+
+```bash
+jq '.[] | {heading, level, tokens, parent_headings}' \
+   rag_storage/kv_store_text_chunks.json | head -30
+```
+
+应观察到以下「规则生效」的迹象：
+
+- 大表前后块的 heading 通常对应 `[part 1]` / `[part n]`（§3.2 表格切片发生）。
+- 细碎条款被合并到接近 `target_ideal` 的块（§3.6 层级合并生效）。
+- `parent_headings` 在不同章节切换处发生跳变，同章节内保持稳定（§3.1 / §3.6 父路径约束）。
+- 大多数 chunk 落在 `[target_ideal, target_max]` 区间（§3.8）；明显偏小的块通常是 `middle` 表格切片（锁定独立）或紧靠章节边界的尾块。
+
+若出现大量低于 `small_tail_threshold` 的尾块，可能是：
+
+- 父标题路径一致性约束过严（不同 `parent_headings` 的相邻小块无法合并，§3.6.1）。
+- 大量 `middle` 表格切片堆积（表格本身就很大）。
+
+### 7.4 常见问题排查
+
+#### 7.4.1 P 没生效，输出与 R 一致
+
+按以下顺序排查：
+
+1. `full_docs[doc_id].process_options` 是否包含 `P`？
+2. `full_docs[doc_id].parse_format` 是否为 `lightrag`？若为 `raw`，说明走的是 legacy 路径，P 会自动降级到 R。
+3. `lightrag_document_path` 指向的 `.blocks.jsonl` 是否存在、是否非空？
+4. 日志中是否有 `paragraph_semantic ... fallback to recursive_character` 字样？
+
+#### 7.4.2 表格被切散、前后说明分离（§3.2 / §3.3 未生效）
+
+- 检查表格是否真的被识别为 `<table format="json">` 或 `<table format="html">`（看 `.blocks.jsonl`）。未识别格式的表格只能走字符切分，无法启动 TableRowSplit 的角色机制。
+- 检查表格 token 数是否真的超过 `table_max`。低于阈值的表格保持完整，不会触发首/中/末切片。
+- 若是连续大表，确认两张表之间的桥接文字是否在**同一 content 行**内 —— 跨 content 行的桥接不参与 B.1 双向重叠。
+
+#### 7.4.3 细碎条款没有被合并（§3.6 未生效）
+
+- 检查相邻条款的 `parent_headings` 是否一致：父标题路径一致性约束会阻止跨主题合并。
+- 检查 `level` 是否一致：同级合并要求相同 `level`，跨级吸收只允许浅吸深。
+- 检查中间是否插入了 `middle` 表格切片：会阻断尾部整批吸收。
+
+#### 7.4.4 出现单个超过 `target_max` 的块
+
+正常情况下 LevelMerge 的真实 token 复测会拒绝超限合并，但以下场景仍可能出现超限块：
+
+- 单行表格自身超过 `target_max`，无锚点可拆，最终走 R 字符切分但单 chunk 仍超限。
+- `enforce_chunk_token_limit_before_embedding` 在 embedding 前会做最后的硬切分，下游不会真把超限 chunk 嵌入向量库。
+
+#### 7.4.5 `[part n]` 后缀异常（§3.4 / §4.4）
+
+- 同一原始 content 行拆出多片但只看到一个 `[part 1]`：检查是否在 LevelMerge 中被合并 —— 合并后保留主块的 part 后缀，不拼接多个。
+- 出现旧式 `[表格片段N]` 后缀：说明使用了旧版 chunker 输出的数据，新版统一为 `[part n]`，需要重新分块。
+
+### 7.5 日志关键字
+
+P 策略相关日志关键字（用于 `grep` 排查）：
+
+- `paragraph_semantic` — 模块入口
+- `fallback to recursive_character` — 整体或单段落降级
+- `table_chunk_role` — 表格角色相关（§3.3）
+- `bridge` — TableBridge 桥接文字处理（§3.3.2）
+- `table_header` / `tables.json` — HeaderRecovery 表头恢复（§3.3.3）
+- `anchor` — AnchorSplit 锚点选择（§3.4）
+
+### 7.6 阶段名 ↔ 规则对照
+
+代码注释、docstring、日志与测试中使用下列**阶段名**作为交叉引用标识。「曾用名」列给出旧版字母编号（仍可能出现在历史 commit / issue / PR 讨论中）：
+
+| 阶段名 | 曾用名 | 对应规则 | 章节 |
+|---|---|---|---|
+| `HeadingBlocks` | Stage A | 标题级基础块 | §3.1 |
+| `TableRowSplit` | Stage B | 表格完整性与行边界切片 | §3.2 |
+| `HeaderRecovery` | Stage B.2 | 切分时为中段/末段切片补回表头 | §3.3.3 |
+| `TableBridge` | Stage B.1 | 连续大表桥接文字双向重叠 | §3.3.2 |
+| `AnchorSplit` | Stage C | 锚点驱动的长块再切分 | §3.4 |
+| `PartLabeling` | Stage C.1 | `[part n]` 行级来源追溯编号 | §4.4 |
+| `HeadingGlue` | Stage D 前置 | 无正文标题粘连 | §3.5 |
+| `LevelMerge` | Stage D | 层级感知的双相位合并 | §3.6 |

docs/ParagraphSemanticChunking.md

@@ -0,0 +1,518 @@
+# Paragraph Semantic Chunking Strategy
+
+## 1. Use Cases and Strategy Selection
+
+### 1.1 What the P Strategy Solves
+
+Paragraph Semantic Chunking (hereafter the **P strategy**) targets documents with a clear sectional structure such as DOCX and PDF. Its core goal: **align chunk boundaries with the document's native semantic boundaries** (headings, paragraphs, table rows) as much as possible, rather than deciding split points purely by token-length counting.
+
+The P strategy mainly addresses these four problems:
+
+1. **Table context fracture**: after a large table is split, the head/tail slices easily detach from their leading explanation, trailing commentary, or intervening bridge text, becoming impossible to understand on their own at recall time.
+2. **Underused hierarchy information**: methods that look only at adjacent paragraphs cannot exploit the parent-heading path or the relationships between same-level clauses.
+3. **Imbalanced fine-grained section sizes**: regulations, standards, and contracts often contain many 100–300 token fine-grained clauses; leaving them unmerged yields chunks that are too short and semantically thin, while merging purely by adjacent length causes cross-topic pollution.
+4. **Long-block re-splitting breaks structure**: when a section is too long, ordinary character splitting ignores table-row boundaries and heading levels.
+
+The P strategy is valid for the structured output of **any parser that can produce a `.blocks.jsonl` sidecar** (`native` / `mineru` / `docling`) — all three persist an identically-structured `.blocks.jsonl` (carrying `heading` / `level` / `parent_headings`) through the shared `write_sidecar()`. Only the `legacy` engine produces no sidecar; input without a sidecar (the legacy path or a parse failure) automatically degrades to the R strategy (see §6).
+
+### 1.2 Comparison of the P / R / V Strategies
+
+| Dimension | R Strategy (Recursive) | V Strategy (SemanticVector) | P Strategy (ParagraphSemantic) |
+|---|---|---|---|
+| Split basis | Cascaded character separators (paragraph → newline → Chinese punctuation → space → character) + token budget | Sentence-level embedding-distance thresholds (percentile / standard deviation / interquartile range / gradient) to find semantic gaps | Heading outline level and `parent_headings` + table-row boundaries + anchors + hierarchy-aware merging |
+| Chunk-size control | `chunk_token_size` hard cap | `chunk_token_size` is only an advisory ceiling; over-limit chunks are re-split via R | `target_max` hard cap + `target_ideal` soft target + table thresholds + tail-absorption threshold acting in concert |
+| Table handling | Table-unaware; may cut in the middle of a table | Table-unaware | Tables under `table_max` stay whole; large tables are sliced along JSON row arrays / HTML `<tr>` row boundaries and re-wrapped as legal `<table>` |
+| Table context | Relies on a window happening to cover it | Relies on embedding distance | First slice glues the leading explanation, last slice glues the trailing commentary, bridge text between consecutive large tables overlaps bidirectionally |
+| Inter-chunk overlap | Global `chunk_overlap_token_size` | No overlap occurs | Section boundaries never overlap; long body text within one section that falls back to R overlaps by `CHUNK_P_OVERLAP_SIZE`; bridge text between consecutive large tables can enter both the preceding and following table chunks |
+| heading metadata | Usually none | Usually none | Inherits or promotes heading; appends a `[part n]` suffix after splitting; preserves `parent_headings` and `level` |
+| Embedding compute cost | None | High (must embed every sentence) | None |
+| Required input | Any text | Any text + an embedding model | Must have a `.blocks.jsonl` sidecar (produced by any of `native` / `mineru` / `docling`), otherwise degrades to R |
+
+### 1.3 How to Choose
+
+| Scenario | Recommended | Reason |
+|---|---|---|
+| Clear section hierarchy (the content-parsing engine must be able to generate a sidecar file) | **P** | Fully exploits heading levels and table-row boundaries; chunk boundaries hug semantics most closely; avoids cross-topic pollution |
+| Document is mostly prose / commentary / long-form body with no clear sectional structure | **V** | Splitting by semantic similarity forms natural boundaries at topic-shift points, more stable than character splitting |
+| Input is plain text, Markdown, code, or logs, or you want the lowest compute cost | **R** | No embedding cost; cascaded separators are robust enough for mixed Chinese/English text |
+| General configuration (file type uncertain) | **R** | P auto-degrades to R when there is no sidecar; V auto-degrades to R when there is no embedding model |
+| Documents with messy heading styles or many pseudo-headings in the body | **R** or **V** | P relies on the parser correctly identifying headings; messy headings shift the basic-block boundaries |
+| Single-row huge tables or unparseable tables | Any | All three strategies ultimately fall back to character level; P still keeps its table-context-gluing advantage |
+
+## 2. Design Goals and Core Invariants
+
+Every rule of the P strategy serves one goal: **align chunk boundaries with the document's native semantic boundaries, and make each chunk understandable on its own at recall time**. It decomposes this goal into concrete rules for three scenarios (tables, long blocks, fine-grained sections), expanded one by one in §3. No matter how the rules combine, the following four **overlap invariants** always hold — they delimit "where text duplication is allowed, and where it is never allowed":
+
+1. **Section boundaries never overlap**: text between different `.blocks.jsonl` content lines is never copied into each other's chunks, avoiding mis-attribution.
+2. **Long body text within a section may overlap**: multiple fragments split from one content line may keep R-style overlap by `chunk_overlap_token_size`, reducing mid-body cuts.
+3. **Bridge text between tables may overlap bidirectionally**: the only cross-paragraph duplication scenario, dedicated to preserving context for consecutive large tables.
+4. **Table rows never overlap each other**: row-level slicing is itself non-overlapping, distinct from R's overlap concept.
+
+### 2.1 Rule-to-Effect Overview
+
+The table below maps each rule of §3 to the effect it achieves and the internal stage that implements it (stage names double as the cross-reference identifiers in code comments, log keywords, and debugging — see §7.6):
+
+| Chunking rule | Effect achieved | Implementing stage | See |
+|---|---|---|---|
+| Heading-level basic chunks | Chunk boundaries align with the document's native structure, not token counts | HeadingBlocks | §3.1 |
+| Table integrity + row-boundary slicing | Tables are not cut mid-cell; slices remain legal `<table>` | TableRowSplit | §3.2 |
+| Table context gluing (roles + bidirectional bridge overlap) | A table's leading explanation, trailing commentary, and bridge text never detach from the table | TableRowSplit / TableBridge | §3.3 |
+| Header recovery (re-attach the header to middle/last slices at split time) | A split table's `middle`/`last` slices keep their column names when recalled alone, without ever exceeding the cap | HeaderRecovery | §3.3.3 |
+| Anchor-driven long-block re-splitting | Over-long sections are split at semantic points, preserving heading levels | AnchorSplit | §3.4 |
+| Body-less heading gluing | A parent heading is never separated from its child content | HeadingGlue | §3.5 |
+| Hierarchy-aware merging | Fine-grained clauses are gathered toward the ideal size without cross-topic pollution | LevelMerge | §3.6 |
+| Overlap rules | Sufficient recall context, yet section/table boundaries are never mis-attributed | Throughout | §3.7 |
+| Size-threshold coordination | Most chunks land in `[target_ideal, target_max]` | Throughout | §3.8 |
+
+### 2.2 Processing Pipeline Overview
+
+The rules above chain into a pipeline that takes `.blocks.jsonl` as input (**each `type == "content"` line is treated as one heading-level basic block**):
+
+```text
+DOCX / PDF / PPTX / …
+  ↓  native(docx) / mineru / docling parser —— emit basic blocks by heading, no token splitting
+.blocks.jsonl + sidecar (.tables.json / .equations.json / .drawings.json / .blocks.assets/)
+  ↓  TableRowSplit: slice oversized tables along row boundaries and assign first/middle/last roles   → §3.2
+  ↓  HeaderRecovery: budget + re-inject the repeating header into middle/last slices during the split → §3.3.3
+  ↓  TableBridge: bidirectional overlap of bridge text between consecutive large tables              → §3.3
+  ↓  AnchorSplit: anchor-driven re-splitting of long text chunks                                     → §3.4
+  ↓  PartLabeling: [part n] line-level provenance numbering (numbered per original content line, hence before cross-line merging)
+  ↓  HeadingGlue: glue body-less heading blocks forward into their strictly-deeper child             → §3.5
+  ↓  LevelMerge: hierarchy-aware two-phase merging                                                   → §3.6
+Final chunk list
+```
+
+## 3. Chunking Rules and Effects
+
+### 3.1 Heading-Level Basic Chunks — Aligning Boundaries with Native Semantics 〔HeadingBlocks〕
+
+**Rule**: each `type == "content"` line of `.blocks.jsonl` is a basic block, i.e. "the body under one heading as one block". Heading identification is performed entirely by the **parser**; **the P chunker itself never scans the document body or judges heading styles**, and it does no token-threshold splitting at parse time.
+
+**Effect**: a chunk's initial boundaries fall naturally on the document outline structure (at heading transitions) rather than at arbitrary token positions; every later stage works on top of this semantically aligned basis.
+
+The three sidecar-producing engines all carve out basic blocks by heading, each obtaining `heading` / `level` / `parent_headings`:
+
+- **native (docx)**: reads `styles.xml`, builds the style-inheritance chain via `<w:basedOn>` to recover the effective `<w:outlineLvl>`; walks the `document.xml` paragraphs resolving the outline level along the chain, mapping original outline levels 0–8 to internal `level` 1–9; maintains a `current_heading_stack`, clearing old headings no shallower than the current level and computing `parent_headings` on each new heading.
+- **mineru**: detects headings by an item's `text_level > 0` or `label` being `title` / `section_header`, using a heading_stack to maintain the parent chain.
+- **docling**: `label="title"` → level 1, `label="section_header"` → `item.level + 1` (default level 2), likewise maintaining the parent chain.
+
+All three ultimately produce a unified `IRBlock` (carrying `heading` / `level` / `parent_headings`), persisted by `write_sidecar()` into an identically-structured `.blocks.jsonl`; tables, equations, and drawings are extracted as single-line tags (`<table id="..." format="json">...</table>` etc.) written to the corresponding sidecar. Every recognizable heading triggers a basic-block boundary, with **no** token-threshold splitting.
+
+The P chunker reads `.blocks.jsonl` directly, treating each content line as an independent processing unit for the subsequent TableRowSplit/AnchorSplit — which also means `[part n]` numbering is **reset independently** per original content line (see §3.4 and §4.4).
+
+### 3.2 Table Integrity and Row-Boundary Slicing — Never Cut a Table Mid-Cell 〔TableRowSplit〕
+
+**Rule**: a table whose token count does not exceed `table_max` **stays whole**; only a table exceeding `table_max` is sliced, and it is **sliced along row boundaries first** — the whole table degrades to character-level splitting only when a slice has collapsed to a single row that still cannot be expressed within the limit.
+
+**Effect**: a table is never cut in the "middle of a cell"; every slice is re-wrapped as a legal `<table>` tag, so downstream parsing and LLM reading can interpret it as a table rather than as broken markup fragments.
+
+#### 3.2.1 Row-Boundary-First Slicing
+
+- `format="json"`: slice along the top-level JSON row array.
+- `format="html"`: slice along `<tr>...</tr>` rows.
+- Tables not explicitly tagged but whose content can be sniffed as JSON / HTML are handled by the same rules.
+
+Before slicing, the `<table {attrs}></table>` wrapper token overhead is debited so that re-wrapped slices stay within `table_max` as much as possible. Each slice is re-wrapped as a legal `<table>` tag for easy downstream parsing.
+
+#### 3.2.2 Row-Level Recursive Re-Slicing
+
+If a row subset still exceeds `table_max` after re-wrapping, it is subdivided further within that row subset. **When a slice has converged to a single row that cannot be kept both `≤ target_max` and header-complete (the row's content itself exceeds the cap, or it fits but leaves no room for the header it would need), the whole table degrades to an R recursive character split of the original `<table>` text (whose body still carries the header), and a `logger.warning` is logged** — the header content survives as plain text along with the original table text and is never silently dropped, nor is a "some `<table>` slices + some orphaned character fragments" mixed output produced. A slice that needs no injected header and whose single row fits `target_max` is still kept whole as legal `<table>` markup. This mechanism keeps table content expressible by row boundaries in legal table structure as much as possible.
+
+#### 3.2.3 Last-Slice Swallow-Back
+
+If a table's last slice has a token count below `table_min_last` and merging it with the previous slice does not exceed `table_max`, the last slice is swallowed back into the previous slice, reducing useless short table chunks.
+
+### 3.3 Table Context Gluing — Leading/Trailing Explanations, Bridges, and Headers Stay Attached 〔TableRowSplit / TableBridge / HeaderRecovery〕
+
+**Rule**: a sliced table glues to surrounding paragraphs differently by "first/middle/last" role; short bridge text between two consecutive large tables is distributed **bidirectionally** to the table chunks on both sides by budget; middle/last slices that lose the header row get the table's repeating header re-injected into their own `<table>` **during the split** (the header's tokens are budgeted out of each slice's cap before splitting).
+
+**Effect**: a table's **leading explanation** enters the first-slice chunk, its **trailing commentary** enters the last-slice chunk, and **bridge text** serves as both the left table's following context and the right table's preceding context — any table slice carries enough context to be understood on its own at recall, with no "table here, explanation in another chunk" fracture. A split table's middle/last slices, even though detached from the first slice that carries the header, get the header row re-injected back at the top of their own `<table>`, so they remain interpretable per-column when recalled alone.
+
+#### 3.3.1 Table Slice Roles and Physical Gluing
+
+Each table slice is given an internal field `table_chunk_role`, and its role determines how it glues to surrounding paragraphs:
+
+| Role | Meaning | Gluing strategy |
+|---|---|---|
+| `first` | The first slice of the original table | Appended to the tail of the current accumulation block, so the table's **leading explanation** enters the same chunk as the first slice |
+| `middle` | A middle slice of the original table | Emitted standalone, avoiding merger with unrelated body text |
+| `last` | The last slice of the original table | Starts a fresh accumulation block, so the **trailing commentary** is automatically appended after the last slice |
+| `none` | A non-table slice or an unsplit whole table | Handled as an ordinary text chunk |
+
+`table_chunk_role` is an internal field that does not survive into the final output, **but it continues to serve as a merging constraint in LevelMerge** (see §3.6.1).
+
+#### 3.3.2 Bidirectional Bridge-Text Overlap Between Consecutive Large Tables 〔TableBridge〕
+
+When the pattern "large table A, short bridge text, large table B" occurs within the same original content line and both tables are split, the bridge text is distributed bidirectionally by context budget:
+
+1. Encode the bridge text into tokens.
+2. Compute the left budget `prev_budget = min(chunk_overlap_token_size, target_max - current token count of the left last slice)`.
+3. Compute the right budget `next_budget = min(chunk_overlap_token_size, target_max - current token count of the right first slice)`.
+4. **If the bridge text fits within both side budgets**: both the left and right table boundary chunks contain the **complete bridge text**.
+5. **If the bridge text is longer**: the prefix enters the left last-slice chunk, the suffix enters the right first-slice chunk; the middle segment exceeding both budgets becomes a standalone ordinary text chunk. This middle chunk **keeps `chunk_overlap_token_size` of R-style overlap with each side**: extending left to re-include the tail of the prefix that went into the left table chunk, and right to include the head of the suffix that went into the right table chunk. Because each side's prefix/suffix is itself ≤ the overlap budget, the overlap span covers the entire prefix and suffix, so **the middle chunk in effect carries the complete bridge text** (the bridge is therefore never fragmented; only its head/tail are **additionally** copied into the neighbouring table chunks). The overlap indices always stay within the bridge tokens, so **`<table>` content is never copied into the middle chunk**.
+
+A single side's budget is further capped at no more than `chunk_token_size / 2`, so bridge text can never dominate the whole chunk.
+
+How this differs from ordinary adjacent chunk overlap:
+
+- Ordinary overlap copies characters or tokens in sequence, regardless of boundary type.
+- The TableBridge mechanism is triggered by table-slice roles, making the bridge text serve simultaneously as the left table's following context and the right table's preceding context, so a bridging explanation is not attributed to only one side's table nor scattered into a separate chunk that is hard to recall.
+
+#### 3.3.3 Header Recovery for Middle/Last Slices 〔HeaderRecovery〕
+
+After a large table is sliced along row boundaries, the header row stays only in the **first slice**; `middle` / `last` slices thus lose the column names and cannot tell each column's meaning when recalled on their own. To fix this, **during TableRowSplit** the header row is re-injected into the non-first slices' own `<table>`, so every slice becomes a complete header-bearing table.
+
+1. **Header source**: at parse time each table's "cross-page repeating header" is written into the sibling `.tables.json` (entry field `table_header`; **only tables that genuinely carry a repeating header have this field**). **The field is stored in the table's own format so merged-cell semantics survive end-to-end**: a `format="json"` table stores it as a JSON 2-D array string (e.g. `[["H1","H2"]]`), a `format="html"` table stores it as the raw `<thead>…</thead>` fragment (preserving `rowspan` / `colspan`). P traces back to the matching table entry via the `id` preserved on the to-be-split `<table>` tag and takes its `table_header`.
+2. **Budgeted reserve, injected at split time**: the header's token cost is reserved out of each slice's body cap **before** splitting (alongside the `<table {attrs}></table>` wrapper overhead). `_split_table_text` splits against that reduced budget, then re-attaches the header into each non-first slice — a `format="json"` slice prepends the header rows to its row array, a `format="html"` slice splices the stored raw `<thead>` fragment **verbatim at the top of the body** (preserving `rowspan` / `colspan` merged-cell semantics, no longer expanding it into a span-less grid); if an HTML slice already carries a `<thead>` (the cut landed inside a multi-row header) injection is skipped to avoid duplication. The slice keeps its original `attrs` (including the leading `id`). Because the room was reserved, **a slice plus its header still stays `≤ target_max`**, so the hard cap is enforced naturally by every downstream stage — there is no late backfill that can overflow the cap. The first slice keeps its own real header row and is not injected again. If a slice has converged to a single row that can no longer hold both the row content and the header within `target_max` (see §3.2.2), **the whole table degrades to an R recursive character split (header included) and a warning is logged** — never leaving an orphaned header-less slice.
+3. **Never fabricate a header** — none of the following are injected: the source table has no `table_header` field in `.tables.json` (no repeating header), `.tables.json` is missing/unreadable, the slice has degraded to a character-level non-`<table>` fragment (no `id` to trace), or the table was not actually split into multiple pieces.
+4. **Format-consistency hard check (corruption raises)**: before injecting, the `table_header`'s format (JSON 2-D array vs `<thead>` fragment) is classified and compared against the to-be-split table's own `format`. A clear disagreement (e.g. an HTML table fed a JSON-array header, or vice versa) means the sidecar is corrupted or mis-attributed, so **`_split_table_text` raises `ValueError` and aborts chunking of that document** rather than emitting a malformed slice with a mismatched header. This is a deliberate "corruption is a hard error" semantic, distinct from rule 3's "a missing header is silently skipped" — a missing header is a tolerable normal case, a format mismatch is a data-corruption signal.
+
+> Because the header enters the slice at split time, a split table's slices are **completely frozen against LevelMerge — never re-merged with each other** (see §3.6.1); otherwise re-merging two slices of one table would duplicate the header mid-body. The recovered header **enters `content`** and counts toward the chunk's token total (headers are typically tiny); it is **not** stored on `heading`.
+
+### 3.4 Anchor-Driven Long-Block Re-Splitting — Cut at Semantic Points, Keep Headings 〔AnchorSplit〕
+
+**Rule**: for content blocks that still exceed `target_max` after TableRowSplit, split in a balanced way at "short-paragraph anchors" first, promoting the chosen anchor to the new heading of the sub-block; when no qualifying anchor exists, fall back through a three-tier "table first → greedy packing → character splitting".
+
+**Effect**: an over-long section is not hard-cut at an arbitrary token position but cut at **natural semantic points** like short subheadings/transition sentences, with sub-blocks inheriting a readable heading and parent-heading path; meanwhile the algorithm **never drops content** and respects the user-configured chunk-size cap as much as possible.
+
+#### 3.4.1 Short-Paragraph Anchors
+
+Recover the content into paragraphs and choose paragraphs satisfying all of the following as candidate anchors:
+
+- The paragraph is not a table (does not start with `<table`).
+- The paragraph text length does not exceed `max_anchor_candidate_length` (100 characters).
+- The paragraph is not the block's first paragraph (so recursion can converge).
+
+#### 3.4.2 Balanced Anchor Selection
+
+Compute the ideal split positions from the target number of sub-blocks, and from the candidate anchors choose the one nearest the ideal position. The chosen anchor is **promoted to the new `heading`** of the following sub-block, and the original heading is written into that sub-block's `parent_headings`.
+
+#### 3.4.3 No-Anchor Fallback
+
+If no qualifying anchor exists:
+
+1. **Table first**: if an over-limit table still exists within the block, invoke TableRowSplit's row-boundary slicing first.
+2. **Greedy packing**: pack the remaining text by paragraph greedily up to near `target_max`.
+3. **Recursive character splitting**: a single over-long ordinary text paragraph degrades to the R strategy (`chunking_by_recursive_character`), using `chunk_overlap_token_size` to keep adjacent text fragments continuous.
+
+The no-anchor fallback path guarantees the algorithm **does not discard content** and respects the user-configured chunk-size cap as much as possible.
+
+### 3.5 Body-Less Heading Gluing — A Parent Heading Never Separates From Its Child 〔HeadingGlue〕
+
+**Rule**: when a block is heading-only (only a heading, no body of its own) and the immediately following block is **strictly deeper**, glue it **forward** into that deeper child block while preserving the shallower **parent-heading** identity; all other cases are left as-is for LevelMerge.
+
+**Effect**: a parent heading like `## 2.4` (no body) is never sliced off as a lone chunk and then absorbed backward by LevelMerge into the previous peer chunk `## 2.3`, becoming separated from its actual child content `### 2.4.1` — the heading always travels with its child content, with no loss of heading-path levels.
+
+Some sections have only a heading and no body of their own (heading-only), e.g.:
+
+```
+## 2.3   Structural dimensions and weight .....   (level 2, has body)
+## 2.4   Environmental adaptability metrics        (level 2, heading-only, no body)
+### 2.4.1   Overview                               (level 3, has body)
+```
+
+If this went straight into LevelMerge, `## 2.4` would become an independent same-level small block and, via Phase A peer merging or batched tail absorption, be **absorbed backward into the tail of the previous peer block `## 2.3`**, separating this parent heading from its actual child content `### 2.4.1`.
+
+A pre-pass (`_glue_heading_only_blocks`) is therefore inserted before LevelMerge. When the current block is heading-only (`content` consists solely of heading lines, detected by `^#{1,6} +`), it **glues forward only**:
+
+- **Trigger**: the immediately following block is **strictly deeper** (greater `level`) and its `table_chunk_role` is `none` or `first`. A `first` slice is "the first slice of a split large table" — when a subsection's body is an oversized table, TableRowSplit's first emitted block has role `first`; the block right after a heading-only line can only be the next line's first emitted block, so its role must be `none` or `first` (`middle`/`last` only occur inside the same line's table).
+- **Keep the `first` role when gluing into a `first` slice**: after gluing `## 2.4` into a `first` slice, the merged block **stays `first`** (the `## 2.4` heading is exactly the preceding context a `first` slice should carry). LevelMerge then will not absorb it backward into `## 2.3` (a `first` slice cannot be absorbed backward), preserving the table-boundary protection; the `none` sub-block behaves exactly as before.
+- **Action**: glue forward into that sub-block, preserving the **parent-heading** identity (`heading` / `level` / `parent_headings` taken from the shallower parent block). That is, `## 2.4` and `### 2.4.1` are bonded into one block, the heading path still centered on `2.4` — sub-block 2.4.1's `parent_headings` already contains 2.4, so hierarchy info is lossless. A chained heading (`# 2` → `## 2.4` → `### 2.4.1`) collapses along the chain, keeping the **shallowest** identity, until the first sub-block with body content is reached.
+- **No backward gluing**: when the next block is **not** deeper (a shallower/sibling heading, or end of list), the heading-only block is left as-is for LevelMerge. It is **not** glued backward into a deeper previous block (e.g. `### 2.3.9`) — absorbing the shallower `## 2.4` heading into a deeper L3 block would invert the hierarchy (deep-absorbs-shallow) and demote the heading's level. Such an orphan heading is handed directly to LevelMerge's normal handling.
+- **Hard cap preserved**: the sub-block came out of AnchorSplit within `target_max`, but prepending the parent heading line(s) can tip it over the cap. Since nothing downstream re-splits an over-limit block (LevelMerge only prevents it from growing further), an over-cap bonded block is re-split here: **first peel off the leading heading line(s)**, split the body at the **full `target_max`** (so later prefix-free body pieces keep the full budget), then glue the heading prefix back onto the **first body piece**. Only when the first body piece is too large to also hold the prefix is it alone re-split with a reduced cap — so a large prefix does not over-fragment the whole subsection. This way the heading always travels with real body content and is never sliced off as a heading-only orphan (which LevelMerge would otherwise absorb backward), and every emitted piece is still ≤ `target_max`. (Degenerate case: when the prefix alone fills the cap — a very long title, or a tiny `chunk_token_size` — it cannot be kept whole, so the whole block is split directly and the oversized heading line is character-split; here the cap wins over heading integrity.)
+- **No extra backfill into the previous block**: because `keep="left"` preserves the parent's `level`, the bonded whole is just an ordinary small block (not pinned as independent). Whether it merges back into the previous block `2.3` follows LevelMerge's existing rules entirely — peer merging when `2.3` is still < `target_ideal`, or tail absorption when the whole is below `small_tail_threshold` (which can pull it even into an already-saturated previous block), both bounded by the re-measured real token ≤ `target_max`. This pre-pass only guarantees the heading is **never detached from its child content**; it does not lock the whole as an independent block — so letting `2.3 + 2.4 + 2.4.1` share one chunk when size allows is exactly the intended anti-fragmentation behaviour.
+
+> Boundary ambiguity: a body line that genuinely begins with `#␠` would be misjudged as a heading line — this is the same heuristic ambiguity already documented and accepted in `lightrag/parser/_markdown.py`, with very low probability in real corpora.
+
+### 3.6 Hierarchy-Aware Merging — Gather Fine-Grained Clauses to the Ideal Size Without Cross-Topic Pollution 〔LevelMerge〕
+
+**Rule**: **process from deeper levels to shallower levels** — first merge same-level small blocks (Phase A), then batch-absorb the tail, finally allow shallow blocks to absorb deep blocks (Phase B); every merge must simultaneously satisfy four constraints: size, table role, level, and parent-heading path.
+
+**Effect**: many 100–300 token fine-grained clauses are merged toward near `target_ideal` (chunks are no longer too short and semantically thin), while **never lumping together adjacent small blocks that belong to different topics / different parent sections** — curing both "chunks too small" and "cross-topic pollution".
+
+#### 3.6.1 Merging Constraints (every merge must satisfy)
+
+1. **Size constraint**: the merged real text token count does not exceed `target_max`; a block that has reached `target_ideal` in principle no longer participates in ordinary same-level merging.
+2. **Role constraint (slice freeze)**: every split-table slice — `first` / `middle` / `last` — is **locked standalone and never participates in any merge** (it neither absorbs forward, nor is absorbed backward, nor joins batched tail absorption). Reason: the repeating header is injected into each slice at TableRowSplit time, so re-merging two slices of one table would duplicate the header mid-body (§3.3.3). A table's boundary explanations were already glued into the first/last slices during the split, so the freeze does not lose context gluing — it only gives up the post-hoc consolidation of small first/last blocks with unrelated neighbours. Only `none` (an ordinary block / an unsplit whole table) may merge.
+3. **Level constraint**: same-level merging happens between equal `level`s; cross-level absorption allows only shallow-absorbs-deep, **forbidding deep from absorbing shallow in reverse**.
+4. **Parent-heading-path consistency constraint**: the key to avoiding cross-topic pollution, with strict semantics by merge direction —
+   - **Same-level merging (Phase A / tail absorption)**: the two blocks' `parent_headings` must be **exactly equal** (true siblings). Blocks with the same `level` but different parent chains (e.g. `2.4.1` and `2.5.1`) may not merge.
+   - **Cross-level absorption (Phase B, shallow-absorbs-deep)**: the deep block must be a **descendant** of the shallow one — the shallow block's full heading path (`parent_headings` + its own `heading`, with any `[part n]` stripped) must be a prefix of the deep block's `parent_headings`. A shallow block absorbing a deep block from a different branch is forbidden.
+   - Blocks with empty `parent_headings` (preamble / non-hierarchical input) are treated as path-compatible and allowed (no hierarchy to pollute).
+
+#### 3.6.2 Phase A: Peer Merging
+
+For adjacent blocks at the current level, when **the current block** is below `target_ideal`, the merged real token count is ≤ `target_max`, and the constraints above are satisfied, merge them into one block (the absorbed neighbour need not be below `target_ideal`; a backward merge additionally requires the previous block to be < `target_ideal`).
+
+Directional rules by table-slice role (all split-table slices are frozen; only `none` may merge):
+
+| Block role | Can absorb the next block forward | Can be absorbed by the previous block |
+|---|:-:|:-:|
+| `none` | Yes | Yes |
+| `first` | No | No |
+| `middle` | No | No |
+| `last` | No | No |
+
+#### 3.6.3 Batched Tail Absorption
+
+If an **ordinary (`none`)** block that has reached `target_ideal` is immediately followed by a run of same-level small blocks whose total token count is below `small_tail_threshold` and whose merged real token count does not exceed `target_max`, then **absorb that run in one shot**. Stop on encountering **any split-table slice** (`first` / `middle` / `last`), or when the parent-heading path diverges; a split-table slice never initiates tail absorption either.
+
+#### 3.6.4 Phase B: Cross-Level Absorption
+
+For small blocks still unsaturated after Phase A, attempt cross-level merging, but allow only shallow-absorbs-deep:
+
+- When the current block is shallower than the next block, the current block may absorb the next block forward.
+- When the current block is deeper than the previous block, the previous shallower block may absorb the current block.
+- Merging in the reverse direction is forbidden.
+- Split-table slices (`first` / `middle` / `last`) are likewise frozen in the cross-level stage and do not participate; only `none` blocks take part in cross-level absorption.
+
+#### 3.6.5 Post-Merge Real-Token Re-Measurement
+
+Because merging inserts a newline joiner, summing per-block token counts may underestimate the merged result. **Before committing every merge, recompute the token count on the joined real text** and confirm it does not exceed `target_max` before committing.
+
+After merging, the main block's `heading` is kept. If multiple part fragments are merged, the final heading keeps the main block's part suffix and does **not** additionally concatenate multiple part tags.
+
+### 3.7 Overlap-Rule Summary — Where It Overlaps, Where It Never Does
+
+**Rule + effect**: the P strategy draws a precise boundary on "text duplication (overlap)", ensuring sufficient recall context while ruling out cross-section/cross-table mis-attribution. The overlap behaviours scattered across stages are gathered here:
+
+| Scenario | Overlaps? | Budget / mechanism | Effect served |
+|---|---|---|---|
+| Different `.blocks.jsonl` content lines (section boundaries) | **Never overlaps** | —— | Clear section boundaries, no mis-attribution |
+| Long body text within one content line falling back to R | May overlap | `chunk_overlap_token_size` | Keeps semantic continuity at a mid-body cut |
+| Bridge text between consecutive large tables | Bidirectional overlap | `min(overlap, …, target_max/2)` per side | Bridge explanation serves as context for both the left and right tables |
+| Standalone middle chunk of a long bridge | Overlaps each side | `chunk_overlap_token_size` (kept within bridge tokens, never includes `<table>`) | The middle reads continuously with the neighbouring table chunks |
+| Between table row-level slices | **Never overlaps** | —— | Row slices are non-overlapping, avoiding duplicate rows |
+
+### 3.8 Size-Threshold Coordination — Most Chunks Land in [ideal, max]
+
+**Rule**: the P strategy's thresholds are not fixed constants but derived dynamically from `chunk_token_size` (denoted N); multiple thresholds act in concert to control the size of text chunks and table slices.
+
+**Effect**: under the ideal distribution, most chunks land in the `[target_ideal, target_max]` interval (about 1500–2000 tokens when N=2000); noticeably small chunks are usually just standalone-locked `middle` table slices or section-boundary tail blocks.
+
+| Name | Formula | Value at N = 2000 | Technical meaning |
+|---|---|---:|---|
+| `target_max` | N | 2000 | Text-chunk hard cap |
+| `target_ideal` | 0.75 × N | 1500 | Text-chunk ideal target; once reached, stops participating in ordinary same-level merging |
+| `table_max` | 0.625 × N | 1250 | Table slicing trigger threshold |
+| `table_ideal` | 0.375 × N | 750 | Table slice ideal size |
+| `table_min_last` | 0.32 × `table_max` | 400 | Table last-slice swallow-back threshold (below this and mergeable → swallowed back into the previous slice) |
+| `small_tail_threshold` | 0.125 × N | 250 | Tail-fragment absorption threshold |
+| `max_anchor_candidate_length` | Fixed | 100 chars | Upper bound on candidate anchor-paragraph length for long-block splitting |
+
+Proportional constraints: `table_max < target_ideal < target_max`, `table_ideal < table_max`. These ratios come from audit-mode empirical values (`large block 8000, small table 5000, ideal table 3000, table tail block 1600`) and are now scaled proportionally by `chunk_token_size`.
+
+## 4. Input and Output
+
+### 4.1 Input
+
+`chunking_by_paragraph_semantic()` accepts the following inputs:
+
+| Parameter | Source | Description |
+|---|---|---|
+| `content` | `full_docs[doc_id].content` | The concatenated merged text, used for degradation when the sidecar is missing |
+| `blocks_path` | `full_docs[doc_id].lightrag_document_path` | The `.blocks.jsonl` path, the P strategy's main input |
+| `.tables.json` (implicit) | Derived from `blocks_path` (`<base>.blocks.jsonl` → `<base>.tables.json`) | The header source for HeaderRecovery (§3.3.3); silently skipped when missing |
+| `chunk_token_size` | `chunk_options.chunk_token_size` / `CHUNK_P_SIZE` | The target hard cap N, default `2000` |
+| `chunk_overlap_token_size` | `CHUNK_P_OVERLAP_SIZE` / `chunk_overlap_token_size` | The cap on long-body fallback within one content line and on the table bridge budget, default `100` |
+| `tokenizer` | The tokenizer already resolved by LightRAG | The basis for all token counting and text-overlap extraction |
+
+The P strategy **does not accept** `split_by_character` / `split_by_character_only`, because the normal path is driven by heading and paragraph structure.
+
+### 4.2 `.blocks.jsonl` Convention
+
+The P strategy only processes `type == "content"` lines. Each content line typically contains:
+
+- `content`: the body text under that heading, possibly containing ordinary paragraphs, `<table ... />` tags, `<equation ... />` formulas, `<drawing ... />` graphics.
+- `heading`: the current heading.
+- `parent_headings`: the parent-heading chain.
+- `level`: the heading level (1–9, corresponding to original outline levels 0–8).
+- `positions`: the original paragraph positions (for traceability).
+- `blockid`: a stable identifier of this content line (optional). When present, it is carried into the final chunk's `sidecar` field, letting the multimodal pipeline and document deletion trace back by source block; when absent (raw / legacy input), the output contains no `sidecar`.
+
+The parser guarantees "the body under one heading as one basic block" (native via heading-driven structural splitting, mineru / docling via their respective IR builders), with no token-threshold splitting at parse time. Tables stay whole, inserted into `content`.
+
+### 4.3 Output
+
+The final output is an ordered chunk list, each element:
+
+```python
+{
+    "tokens": int,                    # Real token count (re-measured after merging)
+    "content": str,                   # Chunk text (may contain <table> tags)
+    "chunk_order_index": int,         # Chunk order index
+    "heading": {                      # Heading metadata (nested dict, not flat fields)
+        "level": int,                 # Heading level
+        "heading": str,               # Gets a [part n] suffix after splitting
+        "parent_headings": list[str], # Parent-heading chain, no suffix appended
+    },
+    # Optional: present only when the input .blocks.jsonl line carries a blockid,
+    # for the multimodal pipeline and document deletion to trace back by source block.
+    "sidecar": {
+        "type": "block",
+        "id": str,                    # Main-block blockid (refs[0])
+        "refs": [{"type": "block", "id": str}, ...],  # All source blockids, deduplicated
+    },
+}
+```
+
+Note: `level` and `parent_headings` are now folded into the nested `heading` dict and are no longer provided at the top level; the `[part n]` suffix lands on `heading["heading"]`. A middle/last table slice's recovered header does not enter `heading`; it is prepended back into the slice's own `<table>` inside the chunk `content` (§3.3.3).
+
+Internally the implementation also uses temporary fields like `paragraphs`, `content`, `table_chunk_role`, `blockids` to aid splitting and merging, but they do **not** enter the final output under those names (`blockids` is materialized as `sidecar` after conversion).
+
+### 4.4 `[part n]` Suffix Rules
+
+- When one original `.blocks.jsonl` content line is split into multiple fragments, every fragment's `heading` field gets `[part 1]`, `[part 2]`, …
+- A content line that was not split keeps its original heading.
+- `parent_headings` gets no suffix.
+- The numbering is **reset independently** within each original content line (because PartLabeling numbers before cross-line merging, see §2.2).
+- The legacy `[表格片段N]` suffix has been unified under `[part n]`.
+
+## 5. Configuration
+
+| Config | Default | Description |
+|---|---|---|
+| `CHUNK_P_SIZE` | `2000` (uses `DEFAULT_CHUNK_P_SIZE` when unset; does **not** inherit `CHUNK_SIZE`) | P-specific `chunk_token_size`; paragraph-semantic merging needs a larger cap than the global default, hence an independent default rather than falling back to `CHUNK_SIZE` |
+| `CHUNK_P_OVERLAP_SIZE` | Unset (inherits `CHUNK_OVERLAP_SIZE`) | P-specific overlap; affects only long-body fallback within one content line and the table bridge budget, and does **not** make table row-level slices overlap each other |
+| `CHUNK_OVERLAP_SIZE` / `LightRAG(chunk_overlap_token_size=…)` | `100` | Global fallback when no P-specific overlap is set |
+
+For config syntax, the precedence chain, runtime overrides via `addon_params["chunker"]`, etc., see [FileProcessingConfiguration-zh.md](FileProcessingConfiguration-zh.md) §3.
+
+`P` is a chunking option orthogonal to the engine (`suffix:engine-options`) and can combine with any sidecar-producing engine. A typical `LIGHTRAG_PARSER` setup enabling P:
+
+```bash
+# docx uses native, pdf uses mineru, other supported formats use docling, all with P; unsupported formats fall back to legacy-R
+LIGHTRAG_PARSER=docx:native-teP,pdf:mineru-iteP,*:docling-iteP,*:legacy-R
+CHUNK_P_SIZE=2000
+CHUNK_P_OVERLAP_SIZE=100
+```
+
+(The option flags `i`/`t`/`e` mean image/table/formula analysis respectively, and `P` is the chunking strategy, combinable as needed.) Or override per file:
+
+```text
+my-proposal.[native-P].docx
+paper.[mineru-P].pdf
+```
+
+## 6. Fallback Protection — Never Drop Content
+
+**Rule + effect**: the P strategy has multi-layer fallback protection; whenever a structural capability fails it retreats to character-level splitting, **guaranteeing the document still produces retrieval chunks and is not silently dropped because a structured sidecar is missing**.
+
+| Trigger | Degradation behaviour |
+|---|---|
+| `blocks_path` missing, unreadable, or with no valid content lines | Degrade wholesale to `chunking_by_recursive_character()`, passing the resolved `chunk_overlap_token_size` |
+| TableRowSplit cannot identify a table's JSON / HTML structure | That table uses R-strategy character splitting |
+| TableRowSplit finds a single row that cannot be kept within `target_max` alongside its header (the row content exceeds the cap, or it fits but the header would push it over) | **The whole table (header included) degrades to R-strategy character splitting and a `logger.warning` is logged**; the header content survives as plain text along with the original table text |
+| AnchorSplit finds a long block with no qualifying short-paragraph anchor | Table first → greedy packing → degrade to R character splitting if a single paragraph is too long |
+| HeaderRecovery finds `.tables.json` missing/unreadable, or the source table has no `table_header` | Skip header injection (that table has no repeating header to begin with; does not affect the rest of chunking) |
+
+**Important**: after a wholesale fallback there is no longer heading hierarchy, table roles, or bidirectional bridge-text overlap; but it guarantees the document still produces retrieval chunks.
+
+## 7. Validating Effects and Debugging
+
+### 7.1 Check Whether the Sidecar Was Generated
+
+Confirm the parser successfully produced `.blocks.jsonl`:
+
+```bash
+ls -l INPUT/__parsed__/<doc>.<ext>.parsed/<doc>.blocks.jsonl
+```
+
+If the file is missing or empty, the P strategy degrades wholesale to R and gains none of P's benefits. Common causes:
+
+- No sidecar-producing engine was configured for that format (e.g. `LIGHTRAG_PARSER=docx:native-...` / `pdf:mineru-...` / `*:docling-...`), so it actually took the `legacy` path.
+- Parse failure (check the `pipeline_status` error entries).
+- The format is not supported by the chosen engine (e.g. native supports only docx; switch to mineru / docling to cover more formats).
+
+### 7.2 Check the blocks.jsonl Content
+
+One JSON per line; after filtering `type == "content"`, inspect whether heading / level / parent_headings match expectations:
+
+```bash
+jq -c 'select(.type=="content") | {level, heading, parent_headings}' \
+   INPUT/__parsed__/<doc>.<ext>.parsed/<doc>.blocks.jsonl | head
+```
+
+If heading is mostly empty or level is abnormal, the parser did not correctly identify headings — in which case the P strategy's hierarchy merging and anchor promotion both fail.
+
+### 7.3 Check Whether the Final Chunks Achieve the Expected Effects
+
+Inspect the chunk metadata in the `text_chunks` store:
+
+```bash
+jq '.[] | {heading, level, tokens, parent_headings}' \
+   rag_storage/kv_store_text_chunks.json | head -30
+```
+
+You should observe the following signs of "rules taking effect":
+
+- The heading of chunks around a large table usually corresponds to `[part 1]` / `[part n]` (§3.2 table slicing happened).
+- Fine-grained clauses are merged into chunks near `target_ideal` (§3.6 hierarchy merging took effect).
+- `parent_headings` jumps at section transitions and stays stable within a section (§3.1 / §3.6 parent-path constraint).
+- Most chunks land in the `[target_ideal, target_max]` interval (§3.8); noticeably small chunks are usually `middle` table slices (locked standalone) or tail blocks right at a section boundary.
+
+If many tail blocks below `small_tail_threshold` appear, it may be:
+
+- The parent-heading-path consistency constraint being too strict (adjacent small blocks with different `parent_headings` cannot merge, §3.6.1).
+- A pile-up of `middle` table slices (the table itself is very large).
+
+### 7.4 Common Troubleshooting
+
+#### 7.4.1 P Did Not Take Effect; Output Matches R
+
+Check in this order:
+
+1. Does `full_docs[doc_id].process_options` include `P`?
+2. Is `full_docs[doc_id].parse_format` equal to `lightrag`? If it is `raw`, the legacy path was taken and P auto-degrades to R.
+3. Does the `.blocks.jsonl` pointed to by `lightrag_document_path` exist and is it non-empty?
+4. Are there `paragraph_semantic ... fallback to recursive_character` lines in the log?
+
+#### 7.4.2 Table Scattered, Leading/Trailing Explanation Separated (§3.2 / §3.3 not in effect)
+
+- Check whether the table was actually identified as `<table format="json">` or `<table format="html">` (look at `.blocks.jsonl`). A table of unrecognized format can only go through character splitting and cannot start TableRowSplit's role mechanism.
+- Check whether the table's token count actually exceeds `table_max`. A table below the threshold stays whole and does not trigger first/middle/last slicing.
+- For consecutive large tables, confirm the bridge text between them is within the **same content line** — a bridge across content lines does not participate in TableBridge bidirectional overlap.
+
+#### 7.4.3 Fine-Grained Clauses Not Merged (§3.6 not in effect)
+
+- Check whether adjacent clauses have consistent `parent_headings`: the parent-heading-path consistency constraint blocks cross-topic merging.
+- Check whether `level` is consistent: same-level merging requires equal `level`, and cross-level absorption allows only shallow-absorbs-deep.
+- Check whether a `middle` table slice is inserted in between: it blocks batched tail absorption.
+
+#### 7.4.4 A Single Chunk Exceeding `target_max` Appears
+
+Normally LevelMerge's real-token re-measurement rejects over-limit merges, but over-limit chunks can still appear in these scenarios:
+
+- A single-row table itself exceeds `target_max`, with no anchor to split on, ultimately going through R character splitting but a single chunk still exceeds the limit.
+- `enforce_chunk_token_limit_before_embedding` does a final hard split before embedding, so downstream never actually embeds an over-limit chunk into the vector store.
+
+#### 7.4.5 `[part n]` Suffix Anomalies (§3.4 / §4.4)
+
+- One original content line was split into multiple pieces but only one `[part 1]` is seen: check whether they were merged in LevelMerge — after merging the main block's part suffix is kept and not concatenated.
+- A legacy `[表格片段N]` suffix appears: this means data output by an old chunker version; the new version unifies on `[part n]`, so re-chunk.
+
+### 7.5 Log Keywords
+
+P-strategy-related log keywords (for `grep` troubleshooting):
+
+- `paragraph_semantic` — module entry
+- `fallback to recursive_character` — wholesale or single-paragraph degradation
+- `table_chunk_role` — table-role related (§3.3)
+- `bridge` — TableBridge bridge-text handling (§3.3.2)
+- `table_header` / `tables.json` — HeaderRecovery header recovery (§3.3.3)
+- `anchor` — AnchorSplit anchor selection (§3.4)
+
+### 7.6 Stage Name ↔ Rule Mapping
+
+The following **stage names** are used as cross-reference identifiers in code comments, docstrings, logs, and tests. The "Former name" column gives the old letter scheme (which may still appear in historical commits / issues / PR discussions):
+
+| Stage name | Former name | Corresponding rule | Section |
+|---|---|---|---|
+| `HeadingBlocks` | Stage A | Heading-level basic chunks | §3.1 |
+| `TableRowSplit` | Stage B | Table integrity and row-boundary slicing | §3.2 |
+| `HeaderRecovery` | Stage B.2 | Re-attach the header to middle/last slices during the split | §3.3.3 |
+| `TableBridge` | Stage B.1 | Bidirectional bridge-text overlap between consecutive large tables | §3.3.2 |
+| `AnchorSplit` | Stage C | Anchor-driven long-block re-splitting | §3.4 |
+| `PartLabeling` | Stage C.1 | `[part n]` line-level provenance numbering | §4.4 |
+| `HeadingGlue` | Stage D pre-pass | Body-less heading gluing | §3.5 |
+| `LevelMerge` | Stage D | Hierarchy-aware two-phase merging | §3.6 |

docs/ParserDebugCLI-zh.md

@@ -0,0 +1,129 @@
+# Parser CLI Debuger使用指南
+
+本工具用于本地调试 LightRAG 注册表中的任意内容解析引擎（内置 `native` / `legacy` / `mineru` / `docling`，以及通过 `lightrag.parsers` entry point 注册的第三方引擎，见 `docs/ThirdPartyParser-zh.md`），针对**单个文件**触发与 pipeline worker 相同的注册表派发路径（`get_parser(engine).parse(...)`），并把解析产物（sidecar 与 raw 缓存）输出到一个**扁平目录布局**——与生产入库目录相比，区别仅在于：
+
+- **无 `__parsed__/` 中间层**：产物直接落在指定父目录下，便于查看；
+- **源文件不会被归档**：源文件保留在原位置（生产路径会把源文件移到 `<INPUT_DIR>/__parsed__/`）；
+- **raw 缓存只看目录是否存在**：`mineru` / `docling` 的 raw 目录非空即视为有效，跳过 `_manifest.json` 校验。
+
+其余流程（IR 构建、sidecar 写入、对 `full_docs` 的同步逻辑）与生产入库完全一致，便于排查解析阶段问题。
+
+## 命令格式
+
+```bash
+python -m lightrag.parser.cli <input_file> \
+    --engine <engine> \
+    [-o <sidecar_parent_dir>] \
+    [--doc-id <doc-id>] \
+    [--force-reparse] \
+    [--preview N]
+```
+
+| 参数 | 说明 |
+|---|---|
+| `input_file` | 待解析的源文件路径（位置参数，必填）。文件必须实际存在。 |
+| `--engine` | 必填，可选值来自注册表：内置 `native`（仅 `.docx`，本地解析）/ `legacy`（纯文本抽取，无 sidecar）/ `mineru`（PDF/办公文档，调 MinerU 服务）/ `docling`（PDF/办公文档，调 docling-serve），以及任何已注册的第三方引擎。 |
+| `-o / --sidecar-parent-dir` | sidecar 与 raw 目录的父目录，默认 = 源文件所在目录。 |
+| `--doc-id` | 自定义文档 ID，默认 `doc-<md5(源文件绝对路径)>`（同一文件多次跑结果稳定）。 |
+| `--force-reparse` | 仅对外部服务引擎（`mineru` / `docling` 及继承 `ExternalParserBase` 的第三方引擎）生效：清空 raw 目录、强制重新下载与解析。默认行为是 raw 目录非空即复用。 |
+| `--preview N` | 解析完成后打印前 N 个 block 的预览（headings + 内容片段），默认 5；`0` 关闭。对无 sidecar 的引擎（如 `legacy`），改为打印解析文本的前 400 字符。 |
+
+## 输出目录布局
+
+以输入 `./inputs/workspace/sample.pdf` + 默认 sidecar 父目录（即 `./inputs/workspace/`）为例：
+
+```
+./inputs/workspace/
+├── sample.pdf                       # 原文件，不动
+├── sample.pdf.parsed/               # ← sidecar 输出
+│   ├── sample.blocks.jsonl          # JSONL：首行 meta，后续每行一个 block
+│   ├── sample.blocks.assets/        # native 抽取的图片/媒体资产（若有）
+│   ├── sample.tables.json           # 表格 sidecar（若 IR 含 tables）
+│   ├── sample.drawings.json         # 图纸/图片 sidecar（若 IR 含 drawings）
+│   └── sample.equations.json        # 公式 sidecar（若 IR 含 equations）
+└── sample.pdf.<engine>_raw/         # ← mineru / docling 的 raw 缓存（native 无此目录）
+    ├── _manifest.json               # 由引擎下载流程写入；CLI 缓存校验不读
+    └── <bundle files>               # 引擎特定 raw 产物（content_list.json / *.json / 资产等）
+```
+
+`native` 引擎不产生 raw 目录（解析是本地的，无外部服务参与）。
+
+## 典型用例
+
+### A. 本地解析 `.docx`（零网络依赖）
+
+```bash
+python -m lightrag.parser.cli ./inputs/workspace/sample.docx --engine native
+# 产出：./inputs/workspace/sample.docx.parsed/  （含 blocks.jsonl + assets）
+```
+
+### B. 用 MinerU 解析 PDF（首次会下载 raw）
+
+```bash
+# 第一次：下载 raw bundle + 生成 sidecar
+python -m lightrag.parser.cli ./inputs/workspace/sample.pdf --engine mineru
+# 第二次（无任何修改）：raw 目录非空 → 直接复用 → 仅重建 sidecar，速度快
+python -m lightrag.parser.cli ./inputs/workspace/sample.pdf --engine mineru
+# 日志会显示： [mineru] raw cache hit doc_id=...
+```
+
+### C. 用 Docling 解析 PDF + 复用已有 raw 目录
+
+```bash
+# 已有 ./inputs/workspace/sample.pdf.docling_raw/ （含 docling 产物的 JSON 等文件）
+python -m lightrag.parser.cli ./inputs/workspace/sample.pdf --engine docling
+# CLI 不查 manifest，只要 raw 目录非空就跳过 docling-serve 调用
+```
+
+> 注：这是旧 `python -m lightrag.parser.external.docling` 调试入口「从已有 raw 重建 sidecar」场景的等价替代——只需把 raw 目录放到约定位置（`<sidecar_parent>/<source>.docling_raw/`）即可触发缓存命中分支。
+
+### D. 输出到自定义目录
+
+```bash
+python -m lightrag.parser.cli ./inputs/workspace/sample.docx \
+    --engine native -o /tmp/debug_sidecar
+# 产出：/tmp/debug_sidecar/sample.docx.parsed/
+# 原文件 ./inputs/workspace/sample.docx 不会被移动
+```
+
+### E. 强制重新解析（清空 raw 后重新下载）
+
+```bash
+python -m lightrag.parser.cli ./inputs/workspace/sample.pdf \
+    --engine docling --force-reparse
+# raw 目录被清空 → 重新调 docling-serve 下载 → 重新生成 sidecar
+```
+
+## 环境变量
+
+`mineru` / `docling` 引擎在 **缓存未命中**（首次解析或 `--force-reparse`）时会调用外部服务，所需环境变量与生产入库一致：
+
+- **MinerU**：`MINERU_API_MODE`（`local` / `official`）、`MINERU_API_TOKEN`、`MINERU_LOCAL_ENDPOINT` 或 `MINERU_OFFICIAL_ENDPOINT`，可选 `MINERU_ENGINE_VERSION` / `MINERU_MODEL_VERSION` / `MINERU_POLL_INTERVAL_SECONDS` / `MINERU_MAX_POLLS`。
+- **Docling**：`DOCLING_ENDPOINT`，可选 `DOCLING_ENGINE_VERSION` / `DOCLING_DO_OCR` / `DOCLING_FORCE_OCR` / `DOCLING_OCR_ENGINE` / `DOCLING_OCR_PRESET` / `DOCLING_OCR_LANG` / `DOCLING_DO_FORMULA_ENRICHMENT` / `DOCLING_POLL_INTERVAL_SECONDS` / `DOCLING_MAX_POLLS`。
+
+详见 [FileProcessingConfiguration-zh.md](./FileProcessingConfiguration-zh.md)。
+
+**缓存命中**时（raw 目录已存在且非空，且未传 `--force-reparse`）无需任何外部服务环境变量——可用于离线复现解析输出。
+
+## 常见排障
+
+| 现象 | 处理 |
+|---|---|
+| `error: input file does not exist: ...` | 检查 `input_file` 路径，必须是已存在的文件（不是 raw 目录）。 |
+| raw 目录存在但 sidecar 内容仍是旧的 | 默认会**复用** raw 重建 sidecar。如果 raw 本身就过期或被替换，加 `--force-reparse` 清空重下。 |
+| MinerU 报 `MINERU_API_TOKEN` 缺失 / Docling 连接 `DOCLING_ENDPOINT` 失败 | 缓存未命中触发了外部服务调用——核对对应环境变量；或确认 raw 目录是否非空（命中缓存时无需服务）。 |
+| 源文件被意外移动 | 不应发生：CLI 已 mock 归档函数。若复现请提 issue（可能是 pipeline 内增加了新的归档调用点）。 |
+| docling 报 `produced zero blocks` | docling raw 中的主 JSON 内容不可解析或为空。检查 raw 目录的 `*.json` 是否合法。 |
+
+## 与生产解析路径的等价性
+
+本 CLI 与 pipeline parse worker 走同一条注册表派发路径——`get_parser(engine).parse(ParseContext(rag, ...))`（`rag` 是 `lightrag/parser/debug.py` 的轻量替身），因此：
+
+- sidecar 字段、命名、内容格式与生产入库完全一致；
+- IR 构建器、`write_sidecar` 调用、`_persist_parsed_full_docs` 行为完全一致；
+- 三处差异均由 CLI 内的 `monkey-patch` 实现，**不修改任何生产代码**：
+  1. `parsed_artifact_dir_for` → 返回扁平路径（无 `__parsed__/`）；
+  2. 解析器实例的 `is_bundle_valid` → 「raw 非空即有效」（仅外部服务引擎）；
+  3. `archive_docx_source_after_full_docs_sync` → no-op，保留源文件。
+
+可与 `tests/parser/docx/golden/native_docx/` 下的 golden fixture 对比验证（CLI 不冻结时间戳，比对时排除 `created_at` 等时间字段即可）。

docs/ParserDebugCLI.md

@@ -0,0 +1,129 @@
+# Parser CLI Debugger Guide
+
+This tool is used to locally debug any parsing engine in LightRAG's registry (the built-in `native` / `legacy` / `mineru` / `docling`, plus third-party engines registered via the `lightrag.parsers` entry point — see `docs/ThirdPartyParser-zh.md`). It drives the same registry dispatch path as the pipeline worker (`get_parser(engine).parse(...)`) for a **single file** and outputs the parsing artifacts (sidecar and raw cache) into a **flat directory layout**. Compared with the production ingestion directory, the only differences are:
+
+- **No `__parsed__/` intermediate layer**: artifacts land directly under the specified parent directory for easy inspection;
+- **The source file is not archived**: the source file stays at its original location (the production path moves the source file to `<INPUT_DIR>/__parsed__/`);
+- **Raw cache validity only checks directory existence**: any non-empty `mineru` / `docling` raw directory is considered valid, skipping `_manifest.json` validation.
+
+The rest of the flow (IR construction, sidecar writing, `full_docs` synchronization logic) is identical to production ingestion, making it convenient for troubleshooting parsing-stage issues.
+
+## Command Format
+
+```bash
+python -m lightrag.parser.cli <input_file> \
+    --engine <engine> \
+    [-o <sidecar_parent_dir>] \
+    [--doc-id <doc-id>] \
+    [--force-reparse] \
+    [--preview N]
+```
+
+| Argument | Description |
+|---|---|
+| `input_file` | Path to the source file to parse (positional argument, required). The file must actually exist. |
+| `--engine` | Required; choices come from the registry: built-in `native` (only `.docx`, local parsing) / `legacy` (plain-text extraction, no sidecar) / `mineru` (PDF/Office documents, calls MinerU service) / `docling` (PDF/Office documents, calls docling-serve), plus any registered third-party engine. |
+| `-o / --sidecar-parent-dir` | Parent directory of the sidecar and raw directories. Defaults to the directory containing the source file. |
+| `--doc-id` | Custom document ID. Defaults to `doc-<md5(absolute path of source file)>` (stable across multiple runs on the same file). |
+| `--force-reparse` | Effective only for external-service engines (`mineru` / `docling` and third-party engines subclassing `ExternalParserBase`): clears the raw directory and forces re-download and re-parse. By default, a non-empty raw directory is reused. |
+| `--preview N` | After parsing completes, prints a preview of the first N blocks (headings + content snippets). Default 5; `0` disables it. For engines without a sidecar (e.g. `legacy`), prints the first 400 characters of the extracted text instead. |
+
+## Output Directory Layout
+
+Taking input `./inputs/workspace/sample.pdf` + the default sidecar parent directory (i.e., `./inputs/workspace/`) as an example:
+
+```
+./inputs/workspace/
+├── sample.pdf                       # original file, untouched
+├── sample.pdf.parsed/               # ← sidecar output
+│   ├── sample.blocks.jsonl          # JSONL: first line is meta, each subsequent line is a block
+│   ├── sample.blocks.assets/        # image/media assets extracted by native (if any)
+│   ├── sample.tables.json           # table sidecar (if IR contains tables)
+│   ├── sample.drawings.json         # drawing/image sidecar (if IR contains drawings)
+│   └── sample.equations.json        # equation sidecar (if IR contains equations)
+└── sample.pdf.<engine>_raw/         # ← raw cache for mineru / docling (native has no such directory)
+    ├── _manifest.json               # written by the engine download flow; not read by CLI cache validation
+    └── <bundle files>               # engine-specific raw artifacts (content_list.json / *.json / assets, etc.)
+```
+
+The `native` engine does not produce a raw directory (parsing is local, with no external service involved).
+
+## Typical Use Cases
+
+### A. Locally parse a `.docx` (zero network dependency)
+
+```bash
+python -m lightrag.parser.cli ./inputs/workspace/sample.docx --engine native
+# Output: ./inputs/workspace/sample.docx.parsed/  (contains blocks.jsonl + assets)
+```
+
+### B. Parse a PDF with MinerU (raw will be downloaded on first run)
+
+```bash
+# First run: download raw bundle + generate sidecar
+python -m lightrag.parser.cli ./inputs/workspace/sample.pdf --engine mineru
+# Second run (no changes): raw directory non-empty → reused directly → only regenerate sidecar, fast
+python -m lightrag.parser.cli ./inputs/workspace/sample.pdf --engine mineru
+# The log will show: [mineru] raw cache hit doc_id=...
+```
+
+### C. Parse a PDF with Docling + reuse an existing raw directory
+
+```bash
+# Existing ./inputs/workspace/sample.pdf.docling_raw/ (contains docling's JSON output, etc.)
+python -m lightrag.parser.cli ./inputs/workspace/sample.pdf --engine docling
+# The CLI does not check the manifest; as long as the raw directory is non-empty, the docling-serve call is skipped
+```
+
+> Note: this is the equivalent replacement for the "rebuild sidecar from an existing raw directory" scenario that used to live in the legacy `python -m lightrag.parser.external.docling` debug entry point — just place the raw directory at the agreed location (`<sidecar_parent>/<source>.docling_raw/`) to trigger the cache-hit branch.
+
+### D. Output to a custom directory
+
+```bash
+python -m lightrag.parser.cli ./inputs/workspace/sample.docx \
+    --engine native -o /tmp/debug_sidecar
+# Output: /tmp/debug_sidecar/sample.docx.parsed/
+# The source file ./inputs/workspace/sample.docx is not moved
+```
+
+### E. Force re-parse (clear raw and re-download)
+
+```bash
+python -m lightrag.parser.cli ./inputs/workspace/sample.pdf \
+    --engine docling --force-reparse
+# raw directory is cleared → docling-serve is called again to download → sidecar regenerated
+```
+
+## Environment Variables
+
+The `mineru` / `docling` engines call external services when the **cache misses** (first parse or `--force-reparse`); the required environment variables are identical to production ingestion:
+
+- **MinerU**: `MINERU_API_MODE` (`local` / `official`), `MINERU_API_TOKEN`, `MINERU_LOCAL_ENDPOINT` or `MINERU_OFFICIAL_ENDPOINT`, optional `MINERU_ENGINE_VERSION` / `MINERU_MODEL_VERSION` / `MINERU_POLL_INTERVAL_SECONDS` / `MINERU_MAX_POLLS`.
+- **Docling**: `DOCLING_ENDPOINT`, optional `DOCLING_ENGINE_VERSION` / `DOCLING_DO_OCR` / `DOCLING_FORCE_OCR` / `DOCLING_OCR_ENGINE` / `DOCLING_OCR_PRESET` / `DOCLING_OCR_LANG` / `DOCLING_DO_FORMULA_ENRICHMENT` / `DOCLING_POLL_INTERVAL_SECONDS` / `DOCLING_MAX_POLLS`.
+
+See [FileProcessingConfiguration.md](./FileProcessingConfiguration.md) for details.
+
+When the **cache is hit** (the raw directory already exists and is non-empty, and `--force-reparse` is not passed), no external service environment variables are needed — this can be used to offline-reproduce parsing output.
+
+## Common Troubleshooting
+
+| Symptom | Action |
+|---|---|
+| `error: input file does not exist: ...` | Check the `input_file` path; it must be an existing file (not a raw directory). |
+| Raw directory exists but sidecar content is still stale | The default behavior is to **reuse** raw and regenerate sidecar. If the raw itself is outdated or has been replaced, add `--force-reparse` to clear and re-download. |
+| MinerU reports `MINERU_API_TOKEN` missing / Docling fails to connect to `DOCLING_ENDPOINT` | A cache miss triggered an external service call — verify the corresponding environment variables; or confirm whether the raw directory is non-empty (no service needed when the cache hits). |
+| Source file is unexpectedly moved | Should not happen: the CLI has mocked the archive function. If reproducible, please file an issue (a new archive call site may have been added in the pipeline). |
+| docling reports `produced zero blocks` | The main JSON content in docling raw is unparseable or empty. Check whether the `*.json` files in the raw directory are valid. |
+
+## Equivalence with the Production Parsing Path
+
+This CLI drives the same registry dispatch path as the pipeline parse worker — `get_parser(engine).parse(ParseContext(rag, ...))` (with `rag` being the lightweight stand-in in `lightrag/parser/debug.py`), so:
+
+- The sidecar fields, naming, and content format are identical to production ingestion;
+- The IR builders, `write_sidecar` calls, and `_persist_parsed_full_docs` behavior are identical;
+- All three differences are implemented via `monkey-patch` inside the CLI — **no production code is modified**:
+  1. `parsed_artifact_dir_for` → returns the flat path (no `__parsed__/`);
+  2. the resolved parser instance's `is_bundle_valid` → "raw is valid if non-empty" (external-service engines only);
+  3. `archive_docx_source_after_full_docs_sync` → no-op, source file preserved.
+
+Results can be cross-validated against golden fixtures under `tests/parser/docx/golden/native_docx/` (the CLI does not freeze timestamps; just exclude time fields such as `created_at` when comparing).

docs/Reproduce.md

@@ -0,0 +1,235 @@
+# Evaluation Result Reproduce
+
+## Dataset
+
+The dataset used in LightRAG can be downloaded from [TommyChien/UltraDomain](https://huggingface.co/datasets/TommyChien/UltraDomain).
+
+## Generate Query
+
+LightRAG uses the following prompt to generate high-level queries, with the corresponding code in `examples/generate_query.py`.
+
+**Prompt**
+
+```
+Given the following description of a dataset:
+
+{description}
+
+Please identify 5 potential users who would engage with this dataset. For each user, list 5 tasks they would perform with this dataset. Then, for each (user, task) combination, generate 5 questions that require a high-level understanding of the entire dataset.
+
+Output the results in the following structure:
+- User 1: [user description]
+    - Task 1: [task description]
+        - Question 1:
+        - Question 2:
+        - Question 3:
+        - Question 4:
+        - Question 5:
+    - Task 2: [task description]
+        ...
+    - Task 5: [task description]
+- User 2: [user description]
+    ...
+- User 5: [user description]
+    ...
+```
+
+## Batch Eval
+
+To evaluate the performance of two RAG systems on high-level queries, LightRAG uses the following prompt, with the specific code available in `reproduce/batch_eval.py`.
+
+**Prompt**
+
+```
+---Role---
+You are an expert tasked with evaluating two answers to the same question based on three criteria: **Comprehensiveness**, **Diversity**, and **Empowerment**.
+---Goal---
+You will evaluate two answers to the same question based on three criteria: **Comprehensiveness**, **Diversity**, and **Empowerment**.
+
+- **Comprehensiveness**: How much detail does the answer provide to cover all aspects and details of the question?
+- **Diversity**: How varied and rich is the answer in providing different perspectives and insights on the question?
+- **Empowerment**: How well does the answer help the reader understand and make informed judgments about the topic?
+
+For each criterion, choose the better answer (either Answer 1 or Answer 2) and explain why. Then, select an overall winner based on these three categories.
+
+Here is the question:
+{query}
+
+Here are the two answers:
+
+**Answer 1:**
+{answer1}
+
+**Answer 2:**
+{answer2}
+
+Evaluate both answers using the three criteria listed above and provide detailed explanations for each criterion.
+
+Output your evaluation in the following JSON format:
+
+{{
+    "Comprehensiveness": {{
+        "Winner": "[Answer 1 or Answer 2]",
+        "Explanation": "[Provide explanation here]"
+    }},
+    "Empowerment": {{
+        "Winner": "[Answer 1 or Answer 2]",
+        "Explanation": "[Provide explanation here]"
+    }},
+    "Overall Winner": {{
+        "Winner": "[Answer 1 or Answer 2]",
+        "Explanation": "[Summarize why this answer is the overall winner based on the three criteria]"
+    }}
+}}
+```
+
+## Overall Performance Table
+
+||**Agriculture**||**CS**||**Legal**||**Mix**||
+|----------------------|---------------|------------|------|------------|---------|------------|-------|------------|
+||NaiveRAG|**LightRAG**|NaiveRAG|**LightRAG**|NaiveRAG|**LightRAG**|NaiveRAG|**LightRAG**|
+|**Comprehensiveness**|32.4%|**67.6%**|38.4%|**61.6%**|16.4%|**83.6%**|38.8%|**61.2%**|
+|**Diversity**|23.6%|**76.4%**|38.0%|**62.0%**|13.6%|**86.4%**|32.4%|**67.6%**|
+|**Empowerment**|32.4%|**67.6%**|38.8%|**61.2%**|16.4%|**83.6%**|42.8%|**57.2%**|
+|**Overall**|32.4%|**67.6%**|38.8%|**61.2%**|15.2%|**84.8%**|40.0%|**60.0%**|
+||RQ-RAG|**LightRAG**|RQ-RAG|**LightRAG**|RQ-RAG|**LightRAG**|RQ-RAG|**LightRAG**|
+|**Comprehensiveness**|31.6%|**68.4%**|38.8%|**61.2%**|15.2%|**84.8%**|39.2%|**60.8%**|
+|**Diversity**|29.2%|**70.8%**|39.2%|**60.8%**|11.6%|**88.4%**|30.8%|**69.2%**|
+|**Empowerment**|31.6%|**68.4%**|36.4%|**63.6%**|15.2%|**84.8%**|42.4%|**57.6%**|
+|**Overall**|32.4%|**67.6%**|38.0%|**62.0%**|14.4%|**85.6%**|40.0%|**60.0%**|
+||HyDE|**LightRAG**|HyDE|**LightRAG**|HyDE|**LightRAG**|HyDE|**LightRAG**|
+|**Comprehensiveness**|26.0%|**74.0%**|41.6%|**58.4%**|26.8%|**73.2%**|40.4%|**59.6%**|
+|**Diversity**|24.0%|**76.0%**|38.8%|**61.2%**|20.0%|**80.0%**|32.4%|**67.6%**|
+|**Empowerment**|25.2%|**74.8%**|40.8%|**59.2%**|26.0%|**74.0%**|46.0%|**54.0%**|
+|**Overall**|24.8%|**75.2%**|41.6%|**58.4%**|26.4%|**73.6%**|42.4%|**57.6%**|
+||GraphRAG|**LightRAG**|GraphRAG|**LightRAG**|GraphRAG|**LightRAG**|GraphRAG|**LightRAG**|
+|**Comprehensiveness**|45.6%|**54.4%**|48.4%|**51.6%**|48.4%|**51.6%**|**50.4%**|49.6%|
+|**Diversity**|22.8%|**77.2%**|40.8%|**59.2%**|26.4%|**73.6%**|36.0%|**64.0%**|
+|**Empowerment**|41.2%|**58.8%**|45.2%|**54.8%**|43.6%|**56.4%**|**50.8%**|49.2%|
+|**Overall**|45.2%|**54.8%**|48.0%|**52.0%**|47.2%|**52.8%**|**50.4%**|49.6%|
+
+## Reproduce
+
+All the code can be found in the `./reproduce` directory.
+
+### Step-0 Extract Unique Contexts
+
+First, extract unique contexts from the datasets.
+
+**Code**
+
+```python
+def extract_unique_contexts(input_directory, output_directory):
+
+    os.makedirs(output_directory, exist_ok=True)
+
+    jsonl_files = glob.glob(os.path.join(input_directory, '*.jsonl'))
+    print(f"Found {len(jsonl_files)} JSONL files.")
+
+    for file_path in jsonl_files:
+        filename = os.path.basename(file_path)
+        name, ext = os.path.splitext(filename)
+        output_filename = f"{name}_unique_contexts.json"
+        output_path = os.path.join(output_directory, output_filename)
+
+        unique_contexts_dict = {}
+
+        print(f"Processing file: {filename}")
+
+        try:
+            with open(file_path, 'r', encoding='utf-8') as infile:
+                for line_number, line in enumerate(infile, start=1):
+                    line = line.strip()
+                    if not line:
+                        continue
+                    try:
+                        json_obj = json.loads(line)
+                        context = json_obj.get('context')
+                        if context and context not in unique_contexts_dict:
+                            unique_contexts_dict[context] = None
+                    except json.JSONDecodeError as e:
+                        print(f"JSON decoding error in file {filename} at line {line_number}: {e}")
+        except FileNotFoundError:
+            print(f"File not found: {filename}")
+            continue
+        except Exception as e:
+            print(f"An error occurred while processing file {filename}: {e}")
+            continue
+
+        unique_contexts_list = list(unique_contexts_dict.keys())
+        print(f"There are {len(unique_contexts_list)} unique `context` entries in the file {filename}.")
+
+        try:
+            with open(output_path, 'w', encoding='utf-8') as outfile:
+                json.dump(unique_contexts_list, outfile, ensure_ascii=False, indent=4)
+            print(f"Unique `context` entries have been saved to: {output_filename}")
+        except Exception as e:
+            print(f"An error occurred while saving to the file {output_filename}: {e}")
+
+    print("All files have been processed.")
+```
+
+### Step-1 Insert Contexts
+
+Insert the extracted contexts into the LightRAG system.
+
+**Code**
+
+```python
+def insert_text(rag, file_path):
+    with open(file_path, mode='r') as f:
+        unique_contexts = json.load(f)
+
+    retries = 0
+    max_retries = 3
+    while retries < max_retries:
+        try:
+            rag.insert(unique_contexts)
+            break
+        except Exception as e:
+            retries += 1
+            print(f"Insertion failed, retrying ({retries}/{max_retries}), error: {e}")
+            time.sleep(10)
+    if retries == max_retries:
+        print("Insertion failed after exceeding the maximum number of retries")
+```
+
+### Step-2 Generate Queries
+
+Extract tokens from the first and second half of each context, then combine them as dataset descriptions to generate queries.
+
+**Code**
+
+```python
+tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
+
+def get_summary(context, tot_tokens=2000):
+    tokens = tokenizer.tokenize(context)
+    half_tokens = tot_tokens // 2
+
+    start_tokens = tokens[1000:1000 + half_tokens]
+    end_tokens = tokens[-(1000 + half_tokens):1000]
+
+    summary_tokens = start_tokens + end_tokens
+    summary = tokenizer.convert_tokens_to_string(summary_tokens)
+
+    return summary
+```
+
+### Step-3 Query
+
+Extract and query LightRAG with the queries generated in Step-2.
+
+**Code**
+
+```python
+def extract_queries(file_path):
+    with open(file_path, 'r') as f:
+        data = f.read()
+
+    data = data.replace('**', '')
+
+    queries = re.findall(r'- Question \d+: (.+)', data)
+
+    return queries
+```

docs/RoleSpecificLLMConfiguration-zh.md

@@ -0,0 +1,376 @@
+# 基于角色的 LLM/VLM 配置指南
+
+LightRAG 支持为不同处理阶段配置不同的 LLM 或 VLM。这个机制适合把低成本模型用于抽取，把更强模型用于最终回答，或为多模态分析单独指定视觉语言模型。
+
+## 角色说明
+
+当前支持四个角色：
+
+| 角色 | 用途 |
+| --- | --- |
+| `EXTRACT` | 文件插入阶段使用的模型，主要复杂实体关系抽取和摘要总结。建议配置关闭思考模式并具备处理复杂问题的高速模型，模型参数量建议为30B或以上，上下文长度至少为32KB。 |
+| `KEYWORD` | 查询阶段关键词抽取，用于检索前的 high-level / low-level keyword 生成。建议配置关闭思考模式的超高速模型，以提高查询阶段的响应速度。模型数量建议为7B或以上。 |
+| `QUERY` | 查询阶段用于根据召回内容最终问答用关乎问题。建议配置开启思考模式的高质量模型。模型能力越强，回答的质量会越高。模型参数量建议为30B或以上，上下文长度至少为32KB。 |
+| `VLM` | 文件插入阶段用于分析图片。需要配置具有图片识别能力的高质量模型。模型参数量建议为30B或以上。 |
+
+如果某个角色没有专门配置，LightRAG 会使用基础 `LLM_*` 配置。
+
+## 基础 LLM 配置
+
+基础配置定义默认 LLM provider、模型、服务地址、认证信息和并发控制：
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_api_key
+
+# 所有 LLM 请求的默认超时时间
+LLM_TIMEOUT=240
+
+# 所有 LLM 调用的默认最大并发数(MAX_ASYNC 作为兼容旧名仍可用)
+MAX_ASYNC_LLM=4
+```
+
+常用字段：
+
+| 变量 | 说明 |
+| --- | --- |
+| `LLM_BINDING` | 基础 LLM provider。支持 `openai`、`ollama`、`lollms`、`azure_openai`、`bedrock`、`gemini`。 |
+| `LLM_MODEL` | 基础模型名。对 Azure OpenAI 通常使用 deployment 名称。 |
+| `LLM_BINDING_HOST` | 基础 provider endpoint。对于 SDK 默认 endpoint，可使用对应 sentinel，例如 `DEFAULT_GEMINI_ENDPOINT` 或 `DEFAULT_BEDROCK_ENDPOINT`。 |
+| `LLM_BINDING_API_KEY` | 基础 API key。Bedrock 不使用这个字段。 |
+| `LLM_TIMEOUT` | 基础 LLM timeout。角色未设置 timeout 时继承它。 |
+| `MAX_ASYNC_LLM` | 基础 LLM 最大并发。角色未设置 `{ROLE}_MAX_ASYNC_LLM` 时继承它。`MAX_ASYNC` 作为兼容旧名仍可用。 |
+
+## 角色覆盖变量
+
+每个角色都可以覆盖 binding、模型、endpoint、API key、并发和 timeout：
+
+```env
+QUERY_LLM_BINDING=openai
+QUERY_LLM_MODEL=gpt-5
+QUERY_LLM_BINDING_HOST=https://api.openai.com/v1
+QUERY_LLM_BINDING_API_KEY=your_query_api_key
+QUERY_MAX_ASYNC_LLM=2
+QUERY_LLM_TIMEOUT=240
+```
+
+变量格式：
+
+| 变量 | 说明 |
+| --- | --- |
+| `{ROLE}_LLM_BINDING` | 覆盖角色 provider。`ROLE` 可为 `EXTRACT`、`KEYWORD`、`QUERY`、`VLM`。 |
+| `{ROLE}_LLM_MODEL` | 覆盖角色模型名。 |
+| `{ROLE}_LLM_BINDING_HOST` | 覆盖角色 endpoint。 |
+| `{ROLE}_LLM_BINDING_API_KEY` | 覆盖角色 API key。Bedrock 不支持。 |
+| `{ROLE}_MAX_ASYNC_LLM` | 覆盖角色最大并发。未设置时继承 `MAX_ASYNC_LLM`。 |
+| `{ROLE}_LLM_TIMEOUT` | 覆盖角色 timeout。未设置时继承 `LLM_TIMEOUT`。 |
+
+## Provider 参数覆盖
+
+provider 细项使用下面的格式：
+
+```env
+{ROLE}_{PROVIDER_PREFIX}_{FIELD}
+```
+
+例如：
+
+```env
+# 只覆盖 QUERY 角色的 OpenAI reasoning effort
+QUERY_OPENAI_LLM_REASONING_EFFORT=medium
+
+# 只覆盖 EXTRACT 角色的 Bedrock 生成参数
+EXTRACT_BEDROCK_LLM_TEMPERATURE=0.0
+EXTRACT_BEDROCK_LLM_MAX_TOKENS=2048
+
+# 只覆盖 VLM 角色的 Gemini 生成参数
+VLM_GEMINI_LLM_MAX_OUTPUT_TOKENS=4096
+VLM_GEMINI_LLM_TEMPERATURE=0.2
+```
+
+常见 provider 前缀：
+
+| Provider | 基础参数前缀 | 角色参数示例 |
+| --- | --- | --- |
+| `openai` / `azure_openai` | `OPENAI_LLM_*` | `QUERY_OPENAI_LLM_REASONING_EFFORT` |
+| `ollama` | `OLLAMA_LLM_*` | `EXTRACT_OLLAMA_LLM_NUM_PREDICT` |
+| `lollms` | 使用 Ollama 兼容参数集合 | `QUERY_OLLAMA_LLM_TEMPERATURE` |
+| `bedrock` | `BEDROCK_LLM_*` | `EXTRACT_BEDROCK_LLM_MAX_TOKENS` |
+| `gemini` | `GEMINI_LLM_*` | `VLM_GEMINI_LLM_THINKING_CONFIG` |
+
+## 继承规则
+
+### 同一个 provider 内覆盖
+
+如果角色没有设置 `{ROLE}_LLM_BINDING`，或设置成与基础 `LLM_BINDING` 相同，角色会继承基础配置：
+
+- 未设置 `{ROLE}_LLM_MODEL` 时继承 `LLM_MODEL`。
+- 未设置 `{ROLE}_LLM_BINDING_HOST` 时继承 `LLM_BINDING_HOST`。
+- 未设置 `{ROLE}_LLM_BINDING_API_KEY` 时继承 `LLM_BINDING_API_KEY`。
+- 未设置 `{ROLE}_LLM_TIMEOUT` 时继承 `LLM_TIMEOUT`。
+- 未设置 `{ROLE}_MAX_ASYNC_LLM` 时继承 `MAX_ASYNC_LLM`。
+- provider 参数先继承基础 provider options，再叠加角色专属 provider options。
+
+因此，同一个 provider 下只想换模型时，只需要写模型名：
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_api_key
+OPENAI_LLM_REASONING_EFFORT=minimal
+
+# QUERY 继承 host、API key、timeout、并发和 OPENAI_LLM_REASONING_EFFORT
+QUERY_LLM_MODEL=gpt-5
+```
+
+### 跨 provider 覆盖
+
+如果角色的 `{ROLE}_LLM_BINDING` 与基础 `LLM_BINDING` 不同，就是跨 provider 配置。当前规则是：
+
+- 必须设置 `{ROLE}_LLM_MODEL`。
+- 非 Bedrock provider 必须设置 `{ROLE}_LLM_BINDING_API_KEY`。
+- 如果没有设置 `{ROLE}_LLM_BINDING_HOST`，LightRAG 会尝试使用该 provider 的默认 host。
+- provider 参数不继承基础 provider options，而是从空配置开始，只叠加角色专属 provider options。
+
+示例：基础使用 Ollama，本地抽取；最终回答改用 OpenAI：
+
+```env
+LLM_BINDING=ollama
+LLM_MODEL=qwen3.5:9b
+LLM_BINDING_HOST=http://localhost:11434
+OLLAMA_LLM_NUM_CTX=32768
+
+QUERY_LLM_BINDING=openai
+QUERY_LLM_MODEL=gpt-5-mini
+QUERY_LLM_BINDING_HOST=https://api.openai.com/v1
+QUERY_LLM_BINDING_API_KEY=your_openai_api_key
+QUERY_OPENAI_LLM_REASONING_EFFORT=minimal
+```
+
+跨 provider 时建议显式设置 `{ROLE}_LLM_BINDING_HOST`，避免默认 host 与基础 provider 的 endpoint 混淆。
+
+### Bedrock 认证规则
+
+Bedrock 不使用 `LLM_BINDING_API_KEY`，也不支持 `{ROLE}_LLM_BINDING_API_KEY`。可用认证方式：
+
+- 全局 SigV4：`AWS_ACCESS_KEY_ID`、`AWS_SECRET_ACCESS_KEY`、`AWS_SESSION_TOKEN`、`AWS_REGION`。
+- 角色级 SigV4：`{ROLE}_AWS_ACCESS_KEY_ID`、`{ROLE}_AWS_SECRET_ACCESS_KEY`、`{ROLE}_AWS_SESSION_TOKEN`、`{ROLE}_AWS_REGION`。
+- 进程级 bearer token：`AWS_BEARER_TOKEN_BEDROCK`。这是 AWS SDK 进程级设置，不能按角色覆盖。
+
+角色级 Bedrock 示例：
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_openai_api_key
+
+EXTRACT_LLM_BINDING=bedrock
+EXTRACT_LLM_MODEL=us.amazon.nova-lite-v1:0
+EXTRACT_LLM_BINDING_HOST=DEFAULT_BEDROCK_ENDPOINT
+EXTRACT_AWS_REGION=us-west-2
+EXTRACT_AWS_ACCESS_KEY_ID=your_extract_access_key
+EXTRACT_AWS_SECRET_ACCESS_KEY=your_extract_secret_key
+EXTRACT_AWS_SESSION_TOKEN=your_optional_session_token
+EXTRACT_BEDROCK_LLM_TEMPERATURE=0.0
+EXTRACT_BEDROCK_LLM_MAX_TOKENS=2048
+```
+
+## Provider 行为对照
+
+| Provider | 角色级 host/base_url | 角色级 API key | 认证限制 |
+| --- | --- | --- | --- |
+| `openai` | 支持，通过 `{ROLE}_LLM_BINDING_HOST` 传给 OpenAI-compatible client。 | 支持 `{ROLE}_LLM_BINDING_API_KEY`，未设置时同 provider 继承基础 `LLM_BINDING_API_KEY`。 | 当前主要是 API key / Bearer 模式。 |
+| `ollama` | 支持，通过 `{ROLE}_LLM_BINDING_HOST` 传给 Ollama client。 | 支持 `{ROLE}_LLM_BINDING_API_KEY`，未设置时同 provider 继承基础 key；底层未收到 key 时会再回退 `OLLAMA_API_KEY`。 | Bearer header。 |
+| `lollms` | 支持，通过 `{ROLE}_LLM_BINDING_HOST` 作为 `base_url`。 | 支持 `{ROLE}_LLM_BINDING_API_KEY`，未设置时同 provider 继承基础 key。 | Bearer header。 |
+| `azure_openai` | 支持，通过 `{ROLE}_LLM_BINDING_HOST` 作为 Azure endpoint。 | 支持 `{ROLE}_LLM_BINDING_API_KEY`，未设置时同 provider 继承基础 key，也可能回退 `AZURE_OPENAI_API_KEY`。 | `AZURE_OPENAI_API_VERSION` 是全局环境变量，不支持角色级覆盖。 |
+| `bedrock` | 支持，通过 `{ROLE}_LLM_BINDING_HOST` 作为 `endpoint_url`；`DEFAULT_BEDROCK_ENDPOINT` 表示交给 AWS SDK 选择。 | 不支持 generic API key。 | 使用全局或角色级 SigV4。`AWS_BEARER_TOKEN_BEDROCK` 是进程级，不能按角色覆盖。 |
+| `gemini` | 支持，通过 `{ROLE}_LLM_BINDING_HOST` 传给 Google GenAI client；`DEFAULT_GEMINI_ENDPOINT` 表示使用 SDK 默认 endpoint。 | AI Studio 模式支持 `{ROLE}_LLM_BINDING_API_KEY`。 | Vertex AI 由 `GOOGLE_GENAI_USE_VERTEXAI`、`GOOGLE_CLOUD_PROJECT`、`GOOGLE_CLOUD_LOCATION`、`GOOGLE_APPLICATION_CREDENTIALS` 控制，都是进程级设置。 |
+
+## 推荐配置模式
+
+### 1. 同 provider 只更换模型
+
+适合用同一个 OpenAI key 和 endpoint，但让最终回答使用更强模型：
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_api_key
+OPENAI_LLM_REASONING_EFFORT=minimal
+
+QUERY_LLM_MODEL=gpt-5
+QUERY_MAX_ASYNC_LLM=2
+```
+
+`QUERY` 会继承基础 host、API key 和 `OPENAI_LLM_REASONING_EFFORT`。
+
+### 2. 同 provider 更换模型并调整参数
+
+适合基础模型用于抽取，最终回答使用更高 reasoning effort：
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_api_key
+OPENAI_LLM_REASONING_EFFORT=minimal
+OPENAI_LLM_MAX_COMPLETION_TOKENS=4096
+
+QUERY_LLM_MODEL=gpt-5
+QUERY_OPENAI_LLM_REASONING_EFFORT=medium
+QUERY_OPENAI_LLM_MAX_COMPLETION_TOKENS=9000
+QUERY_LLM_TIMEOUT=240
+```
+
+### 3. 同 provider 使用不同 endpoint 和 API key
+
+适合所有角色都走 `openai` binding，但其中一些角色访问 OpenAI 官方接口，另一些角色访问本地 vLLM、SGLang 或 OpenRouter 等 OpenAI-compatible endpoint。下面的例子中：
+
+- `EXTRACT` 使用 OpenAI 官方 `gpt-5-mini`。
+- `QUERY` 使用 OpenAI 官方 `gpt-5.4`，并使用单独的 OpenAI key。
+- `KEYWORD` 使用本地 vLLM 部署的 `Qwen3.5-35B-A3B`。
+
+```env
+###########################################################################
+# Base LLM fallback. Keep it aligned with EXTRACT so unspecified roles still
+# have a valid OpenAI configuration.
+###########################################################################
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_extract_openai_api_key
+LLM_TIMEOUT=240
+MAX_ASYNC_LLM=4
+
+###########################################################################
+# IMPORTANT:
+# Do not set global OPENAI_LLM_REASONING_EFFORT here if any same-provider role
+# points to a local OpenAI-compatible server that does not support it.
+# Use role-specific OPENAI options instead.
+###########################################################################
+# OPENAI_LLM_REASONING_EFFORT=none
+
+###########################################################################
+# EXTRACT: OpenAI official API, gpt-5-mini
+###########################################################################
+EXTRACT_LLM_BINDING=openai
+EXTRACT_LLM_MODEL=gpt-5-mini
+EXTRACT_LLM_BINDING_HOST=https://api.openai.com/v1
+EXTRACT_LLM_BINDING_API_KEY=your_extract_openai_api_key
+EXTRACT_OPENAI_LLM_REASONING_EFFORT=low
+EXTRACT_OPENAI_LLM_MAX_COMPLETION_TOKENS=4096
+EXTRACT_MAX_ASYNC_LLM=4
+EXTRACT_LLM_TIMEOUT=180
+
+###########################################################################
+# QUERY: OpenAI official API, gpt-5.4, separate API key
+###########################################################################
+QUERY_LLM_BINDING=openai
+QUERY_LLM_MODEL=gpt-5.4
+QUERY_LLM_BINDING_HOST=https://api.openai.com/v1
+QUERY_LLM_BINDING_API_KEY=your_query_openai_api_key
+QUERY_OPENAI_LLM_REASONING_EFFORT=medium
+QUERY_OPENAI_LLM_MAX_COMPLETION_TOKENS=9000
+QUERY_MAX_ASYNC_LLM=2
+QUERY_LLM_TIMEOUT=240
+
+###########################################################################
+# KEYWORD: local vLLM OpenAI-compatible endpoint, Qwen3.5-35B-A3B
+###########################################################################
+KEYWORD_LLM_BINDING=openai
+KEYWORD_LLM_MODEL=Qwen3.5-35B-A3B
+KEYWORD_LLM_BINDING_HOST=http://localhost:8000/v1
+# If vLLM was started with --api-key, use the same value here.
+# If vLLM has no auth, still set a non-empty dummy value to avoid falling
+# back to the official OpenAI key.
+KEYWORD_LLM_BINDING_API_KEY=local-vllm-api-key
+KEYWORD_OPENAI_LLM_MAX_TOKENS=2048
+# Optional for Qwen-style models served by vLLM when you want to disable thinking.
+KEYWORD_OPENAI_LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'
+KEYWORD_MAX_ASYNC_LLM=4
+KEYWORD_LLM_TIMEOUT=60
+```
+
+这个模式不是跨 provider，因为三个角色的 binding 都是 `openai`。LightRAG 会分别把每个角色的 `*_LLM_BINDING_HOST` 和 `*_LLM_BINDING_API_KEY` 传给 OpenAI-compatible client。
+
+注意：同 provider 的 provider options 会继承基础 `OPENAI_LLM_*`。如果本地 vLLM 不支持 OpenAI 官方参数，例如 `reasoning_effort`，不要设置全局 `OPENAI_LLM_REASONING_EFFORT`；改用 `EXTRACT_OPENAI_LLM_REASONING_EFFORT`、`QUERY_OPENAI_LLM_REASONING_EFFORT` 这类角色级变量。
+
+### 4. 某个角色跨 provider
+
+适合基础使用 OpenAI 官方模型，只有关键词抽取使用本地 Ollama：
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_openai_api_key
+OPENAI_LLM_REASONING_EFFORT=medium
+
+KEYWORD_LLM_BINDING=ollama
+KEYWORD_LLM_MODEL=qwen3.5:9b
+KEYWORD_LLM_BINDING_HOST=http://localhost:11434
+KEYWORD_LLM_BINDING_API_KEY=ollama-local-key
+KEYWORD_OLLAMA_LLM_NUM_CTX=32768
+```
+
+跨 provider 时，Ollama 参数不会继承 OpenAI 参数。`KEYWORD_LLM_BINDING_API_KEY` 对本地 Ollama 通常可以使用占位值；当前跨 provider 校验会要求非 Bedrock 角色显式提供角色级 API key。
+
+### 5. 为 VLM 单独指定多模态模型
+
+适合文本任务使用便宜模型，多模态分析使用视觉语言模型：
+
+```env
+VLM_PROCESS_ENABLE=true
+
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_api_key
+
+VLM_LLM_BINDING=openai
+VLM_LLM_MODEL=gpt-4o
+VLM_OPENAI_LLM_MAX_TOKENS=4096
+VLM_MAX_ASYNC_LLM=2
+VLM_LLM_TIMEOUT=240
+```
+
+如果 VLM 使用同一个 provider 和 key，可以省略 `VLM_LLM_BINDING_HOST` 与 `VLM_LLM_BINDING_API_KEY`。
+
+`VLM_PROCESS_ENABLE` 是多模态分析的总开关：设为 `false` 时，pipeline 会对每个多模态 item 输出 warning 并跳过，不调用 VLM；设为 `true` 时，生效的 VLM binding（设置了 `VLM_LLM_BINDING` 时取该值，否则取 `LLM_BINDING`）必须支持图片输入。当前支持视觉输入的 provider 包括：`openai`、`azure_openai`、`gemini`、`bedrock`、`ollama`、`anthropic`。`lollms` 无法接收图片输入，会在启动时直接报错。
+
+### 6. Bedrock 角色级 SigV4 凭证
+
+适合只有某个角色访问 Bedrock，并使用独立 IAM/STS 凭证：
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_openai_api_key
+
+QUERY_LLM_BINDING=bedrock
+QUERY_LLM_MODEL=us.amazon.nova-lite-v1:0
+QUERY_LLM_BINDING_HOST=DEFAULT_BEDROCK_ENDPOINT
+QUERY_AWS_REGION=us-east-1
+QUERY_AWS_ACCESS_KEY_ID=your_query_access_key
+QUERY_AWS_SECRET_ACCESS_KEY=your_query_secret_key
+QUERY_AWS_SESSION_TOKEN=your_optional_session_token
+QUERY_BEDROCK_LLM_MAX_TOKENS=4096
+QUERY_BEDROCK_LLM_TEMPERATURE=0.2
+```
+
+不要设置 `QUERY_LLM_BINDING_API_KEY`，Bedrock 会拒绝该配置。
+
+## 注意事项
+
+- 同 provider 下，`OPENAI_LLM_REASONING_EFFORT`、`OPENAI_LLM_MAX_TOKENS`、`OLLAMA_LLM_NUM_CTX`、`GEMINI_LLM_THINKING_CONFIG` 等 provider 参数会自动继承。
+- 当前没有干净的角色级“取消继承某个 provider 参数”的语义。如果某个同 provider 角色模型不支持基础参数，需要为该角色显式覆盖为可用值，或将它配置成跨 provider，并且只设置该角色支持的 provider 参数。
+- `azure_openai` 的 `AZURE_OPENAI_DEPLOYMENT` 和 `AZURE_OPENAI_API_VERSION` 是全局环境变量。若设置了 `AZURE_OPENAI_DEPLOYMENT`，它可能优先于角色模型名。
+- Gemini Vertex AI 模式由进程级 Google 环境变量控制，不能在同一个 LightRAG 进程里让某些角色使用 Vertex AI、另一些角色使用 AI Studio API key。
+- `LLM_BINDING_HOST` 在 Docker/Compose 中通常需要使用容器可访问地址，例如 `host.docker.internal`，角色级 host 也遵循相同原则。
+- 修改 `.env` 后请重启 LightRAG Server。部分 IDE 终端会预加载 `.env`，建议打开新的终端会话确认环境变量生效。

docs/RoleSpecificLLMConfiguration.md

@@ -0,0 +1,376 @@
+# Role-Specific LLM/VLM Configuration Guide
+
+LightRAG supports configuring different LLMs or VLMs for different processing stages. This mechanism is useful when using a lower-cost model for extraction, a stronger model for final answers, or a dedicated vision-language model for multimodal analysis.
+
+## Role Overview
+
+Four roles are currently supported:
+
+| Role | Purpose |
+| --- | --- |
+| `EXTRACT` | The model used during the file insertion stage, mainly for complex entity/relation extraction and summarization. A fast model with thinking mode disabled and the ability to handle complex problems is recommended; a parameter size of 30B or above and a context length of at least 32KB are suggested. |
+| `KEYWORD` | Query-stage keyword extraction for high-level / low-level keyword generation before retrieval. An ultra-fast model with thinking mode disabled is recommended to improve query-stage response speed; a parameter size of 7B or above is suggested. |
+| `QUERY` | The query stage, used to produce the final answer to the question based on the recalled content. A high-quality model with thinking mode enabled is recommended; the stronger the model, the higher the answer quality. A parameter size of 30B or above and a context length of at least 32KB are suggested. |
+| `VLM` | Used during the file insertion stage to analyze images. A high-quality model with image recognition capability is required; a parameter size of 30B or above is suggested. |
+
+If a role has no dedicated configuration, LightRAG uses the base `LLM_*` configuration.
+
+## Base LLM Configuration
+
+The base configuration defines the default LLM provider, model, service endpoint, authentication information, and concurrency control:
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_api_key
+
+# Default timeout for all LLM requests
+LLM_TIMEOUT=240
+
+# Default maximum concurrency for all LLM calls (MAX_ASYNC is still accepted as a deprecated alias)
+MAX_ASYNC_LLM=4
+```
+
+Common fields:
+
+| Variable | Description |
+| --- | --- |
+| `LLM_BINDING` | Base LLM provider. Supported values are `openai`, `ollama`, `lollms`, `azure_openai`, `bedrock`, and `gemini`. |
+| `LLM_MODEL` | Base model name. For Azure OpenAI, this is usually the deployment name. |
+| `LLM_BINDING_HOST` | Base provider endpoint. For SDK default endpoints, use the corresponding sentinel, such as `DEFAULT_GEMINI_ENDPOINT` or `DEFAULT_BEDROCK_ENDPOINT`. |
+| `LLM_BINDING_API_KEY` | Base API key. Bedrock does not use this field. |
+| `LLM_TIMEOUT` | Base LLM timeout. A role inherits it when no role timeout is set. |
+| `MAX_ASYNC_LLM` | Base maximum LLM concurrency. A role inherits it when `{ROLE}_MAX_ASYNC_LLM` is not set. `MAX_ASYNC` is still accepted as a deprecated alias. |
+
+## Role Override Variables
+
+Each role can override the binding, model, endpoint, API key, concurrency, and timeout:
+
+```env
+QUERY_LLM_BINDING=openai
+QUERY_LLM_MODEL=gpt-5
+QUERY_LLM_BINDING_HOST=https://api.openai.com/v1
+QUERY_LLM_BINDING_API_KEY=your_query_api_key
+QUERY_MAX_ASYNC_LLM=2
+QUERY_LLM_TIMEOUT=240
+```
+
+Variable format:
+
+| Variable | Description |
+| --- | --- |
+| `{ROLE}_LLM_BINDING` | Overrides the role provider. `ROLE` can be `EXTRACT`, `KEYWORD`, `QUERY`, or `VLM`. |
+| `{ROLE}_LLM_MODEL` | Overrides the role model name. |
+| `{ROLE}_LLM_BINDING_HOST` | Overrides the role endpoint. |
+| `{ROLE}_LLM_BINDING_API_KEY` | Overrides the role API key. Bedrock does not support it. |
+| `{ROLE}_MAX_ASYNC_LLM` | Overrides the role maximum concurrency. Inherits `MAX_ASYNC_LLM` when unset. |
+| `{ROLE}_LLM_TIMEOUT` | Overrides the role timeout. Inherits `LLM_TIMEOUT` when unset. |
+
+## Provider Option Overrides
+
+Provider-specific options use the following format:
+
+```env
+{ROLE}_{PROVIDER_PREFIX}_{FIELD}
+```
+
+Examples:
+
+```env
+# Override only the OpenAI reasoning effort for the QUERY role
+QUERY_OPENAI_LLM_REASONING_EFFORT=medium
+
+# Override only Bedrock generation parameters for the EXTRACT role
+EXTRACT_BEDROCK_LLM_TEMPERATURE=0.0
+EXTRACT_BEDROCK_LLM_MAX_TOKENS=2048
+
+# Override only Gemini generation parameters for the VLM role
+VLM_GEMINI_LLM_MAX_OUTPUT_TOKENS=4096
+VLM_GEMINI_LLM_TEMPERATURE=0.2
+```
+
+Common provider prefixes:
+
+| Provider | Base option prefix | Role option example |
+| --- | --- | --- |
+| `openai` / `azure_openai` | `OPENAI_LLM_*` | `QUERY_OPENAI_LLM_REASONING_EFFORT` |
+| `ollama` | `OLLAMA_LLM_*` | `EXTRACT_OLLAMA_LLM_NUM_PREDICT` |
+| `lollms` | Uses the Ollama-compatible option set | `QUERY_OLLAMA_LLM_TEMPERATURE` |
+| `bedrock` | `BEDROCK_LLM_*` | `EXTRACT_BEDROCK_LLM_MAX_TOKENS` |
+| `gemini` | `GEMINI_LLM_*` | `VLM_GEMINI_LLM_THINKING_CONFIG` |
+
+## Inheritance Rules
+
+### Overrides Within the Same Provider
+
+If a role does not set `{ROLE}_LLM_BINDING`, or sets it to the same value as the base `LLM_BINDING`, the role inherits the base configuration:
+
+- Inherits `LLM_MODEL` when `{ROLE}_LLM_MODEL` is not set.
+- Inherits `LLM_BINDING_HOST` when `{ROLE}_LLM_BINDING_HOST` is not set.
+- Inherits `LLM_BINDING_API_KEY` when `{ROLE}_LLM_BINDING_API_KEY` is not set.
+- Inherits `LLM_TIMEOUT` when `{ROLE}_LLM_TIMEOUT` is not set.
+- Inherits `MAX_ASYNC_LLM` when `{ROLE}_MAX_ASYNC_LLM` is not set.
+- Provider options first inherit the base provider options, then apply role-specific provider options.
+
+Therefore, when you only want to change the model within the same provider, you only need to set the model name:
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_api_key
+OPENAI_LLM_REASONING_EFFORT=minimal
+
+# QUERY inherits host, API key, timeout, concurrency, and OPENAI_LLM_REASONING_EFFORT
+QUERY_LLM_MODEL=gpt-5
+```
+
+### Cross-Provider Overrides
+
+If a role's `{ROLE}_LLM_BINDING` differs from the base `LLM_BINDING`, it is a cross-provider configuration. The current rules are:
+
+- `{ROLE}_LLM_MODEL` must be set.
+- Non-Bedrock providers must set `{ROLE}_LLM_BINDING_API_KEY`.
+- If `{ROLE}_LLM_BINDING_HOST` is not set, LightRAG tries to use that provider's default host.
+- Provider options do not inherit base provider options. They start empty and only apply role-specific provider options.
+
+Example: use Ollama as the base for local extraction, then use OpenAI for final answers:
+
+```env
+LLM_BINDING=ollama
+LLM_MODEL=qwen3.5:9b
+LLM_BINDING_HOST=http://localhost:11434
+OLLAMA_LLM_NUM_CTX=32768
+
+QUERY_LLM_BINDING=openai
+QUERY_LLM_MODEL=gpt-5-mini
+QUERY_LLM_BINDING_HOST=https://api.openai.com/v1
+QUERY_LLM_BINDING_API_KEY=your_openai_api_key
+QUERY_OPENAI_LLM_REASONING_EFFORT=minimal
+```
+
+For cross-provider configurations, explicitly setting `{ROLE}_LLM_BINDING_HOST` is recommended to avoid confusion between the default host and the base provider endpoint.
+
+### Bedrock Authentication Rules
+
+Bedrock does not use `LLM_BINDING_API_KEY` and does not support `{ROLE}_LLM_BINDING_API_KEY`. Available authentication methods are:
+
+- Global SigV4: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, and `AWS_REGION`.
+- Role-level SigV4: `{ROLE}_AWS_ACCESS_KEY_ID`, `{ROLE}_AWS_SECRET_ACCESS_KEY`, `{ROLE}_AWS_SESSION_TOKEN`, and `{ROLE}_AWS_REGION`.
+- Process-level bearer token: `AWS_BEARER_TOKEN_BEDROCK`. This is an AWS SDK process-level setting and cannot be overridden per role.
+
+Role-level Bedrock example:
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_openai_api_key
+
+EXTRACT_LLM_BINDING=bedrock
+EXTRACT_LLM_MODEL=us.amazon.nova-lite-v1:0
+EXTRACT_LLM_BINDING_HOST=DEFAULT_BEDROCK_ENDPOINT
+EXTRACT_AWS_REGION=us-west-2
+EXTRACT_AWS_ACCESS_KEY_ID=your_extract_access_key
+EXTRACT_AWS_SECRET_ACCESS_KEY=your_extract_secret_key
+EXTRACT_AWS_SESSION_TOKEN=your_optional_session_token
+EXTRACT_BEDROCK_LLM_TEMPERATURE=0.0
+EXTRACT_BEDROCK_LLM_MAX_TOKENS=2048
+```
+
+## Provider Behavior Matrix
+
+| Provider | Role-level host/base_url | Role-level API key | Authentication limitations |
+| --- | --- | --- | --- |
+| `openai` | Supported, passed to the OpenAI-compatible client through `{ROLE}_LLM_BINDING_HOST`. | Supports `{ROLE}_LLM_BINDING_API_KEY`; when unset within the same provider, it inherits the base `LLM_BINDING_API_KEY`. | Currently mainly API key / Bearer mode. |
+| `ollama` | Supported, passed to the Ollama client through `{ROLE}_LLM_BINDING_HOST`. | Supports `{ROLE}_LLM_BINDING_API_KEY`; when unset within the same provider, it inherits the base key. If no key reaches the lower layer, it falls back to `OLLAMA_API_KEY`. | Bearer header. |
+| `lollms` | Supported, using `{ROLE}_LLM_BINDING_HOST` as `base_url`. | Supports `{ROLE}_LLM_BINDING_API_KEY`; when unset within the same provider, it inherits the base key. | Bearer header. |
+| `azure_openai` | Supported, using `{ROLE}_LLM_BINDING_HOST` as the Azure endpoint. | Supports `{ROLE}_LLM_BINDING_API_KEY`; when unset within the same provider, it inherits the base key and may also fall back to `AZURE_OPENAI_API_KEY`. | `AZURE_OPENAI_API_VERSION` is a global environment variable and does not support role-level overrides. |
+| `bedrock` | Supported, using `{ROLE}_LLM_BINDING_HOST` as `endpoint_url`; `DEFAULT_BEDROCK_ENDPOINT` means letting the AWS SDK choose. | Generic API keys are not supported. | Uses global or role-level SigV4. `AWS_BEARER_TOKEN_BEDROCK` is process-level and cannot be overridden per role. |
+| `gemini` | Supported, passed to the Google GenAI client through `{ROLE}_LLM_BINDING_HOST`; `DEFAULT_GEMINI_ENDPOINT` means using the SDK default endpoint. | AI Studio mode supports `{ROLE}_LLM_BINDING_API_KEY`. | Vertex AI is controlled by `GOOGLE_GENAI_USE_VERTEXAI`, `GOOGLE_CLOUD_PROJECT`, `GOOGLE_CLOUD_LOCATION`, and `GOOGLE_APPLICATION_CREDENTIALS`; all are process-level settings. |
+
+## Recommended Configuration Patterns
+
+### 1. Same Provider, Only Change the Model
+
+Suitable when using the same OpenAI key and endpoint, but using a stronger model for final answers:
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_api_key
+OPENAI_LLM_REASONING_EFFORT=minimal
+
+QUERY_LLM_MODEL=gpt-5
+QUERY_MAX_ASYNC_LLM=2
+```
+
+`QUERY` inherits the base host, API key, and `OPENAI_LLM_REASONING_EFFORT`.
+
+### 2. Same Provider, Change the Model and Tune Options
+
+Suitable when the base model is used for extraction and final answers use a higher reasoning effort:
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_api_key
+OPENAI_LLM_REASONING_EFFORT=minimal
+OPENAI_LLM_MAX_COMPLETION_TOKENS=4096
+
+QUERY_LLM_MODEL=gpt-5
+QUERY_OPENAI_LLM_REASONING_EFFORT=medium
+QUERY_OPENAI_LLM_MAX_COMPLETION_TOKENS=9000
+QUERY_LLM_TIMEOUT=240
+```
+
+### 3. Same Provider with Different Endpoints and API Keys
+
+Suitable when all roles use the `openai` binding, but some roles access the official OpenAI API while others access a local vLLM, SGLang, OpenRouter, or another OpenAI-compatible endpoint. In the example below:
+
+- `EXTRACT` uses the official OpenAI `gpt-5-mini`.
+- `QUERY` uses the official OpenAI `gpt-5.4` with a separate OpenAI key.
+- `KEYWORD` uses `Qwen3.5-35B-A3B` deployed by local vLLM.
+
+```env
+###########################################################################
+# Base LLM fallback. Keep it aligned with EXTRACT so unspecified roles still
+# have a valid OpenAI configuration.
+###########################################################################
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_extract_openai_api_key
+LLM_TIMEOUT=240
+MAX_ASYNC_LLM=4
+
+###########################################################################
+# IMPORTANT:
+# Do not set global OPENAI_LLM_REASONING_EFFORT here if any same-provider role
+# points to a local OpenAI-compatible server that does not support it.
+# Use role-specific OPENAI options instead.
+###########################################################################
+# OPENAI_LLM_REASONING_EFFORT=none
+
+###########################################################################
+# EXTRACT: OpenAI official API, gpt-5-mini
+###########################################################################
+EXTRACT_LLM_BINDING=openai
+EXTRACT_LLM_MODEL=gpt-5-mini
+EXTRACT_LLM_BINDING_HOST=https://api.openai.com/v1
+EXTRACT_LLM_BINDING_API_KEY=your_extract_openai_api_key
+EXTRACT_OPENAI_LLM_REASONING_EFFORT=low
+EXTRACT_OPENAI_LLM_MAX_COMPLETION_TOKENS=4096
+EXTRACT_MAX_ASYNC_LLM=4
+EXTRACT_LLM_TIMEOUT=180
+
+###########################################################################
+# QUERY: OpenAI official API, gpt-5.4, separate API key
+###########################################################################
+QUERY_LLM_BINDING=openai
+QUERY_LLM_MODEL=gpt-5.4
+QUERY_LLM_BINDING_HOST=https://api.openai.com/v1
+QUERY_LLM_BINDING_API_KEY=your_query_openai_api_key
+QUERY_OPENAI_LLM_REASONING_EFFORT=medium
+QUERY_OPENAI_LLM_MAX_COMPLETION_TOKENS=9000
+QUERY_MAX_ASYNC_LLM=2
+QUERY_LLM_TIMEOUT=240
+
+###########################################################################
+# KEYWORD: local vLLM OpenAI-compatible endpoint, Qwen3.5-35B-A3B
+###########################################################################
+KEYWORD_LLM_BINDING=openai
+KEYWORD_LLM_MODEL=Qwen3.5-35B-A3B
+KEYWORD_LLM_BINDING_HOST=http://localhost:8000/v1
+# If vLLM was started with --api-key, use the same value here.
+# If vLLM has no auth, still set a non-empty dummy value to avoid falling
+# back to the official OpenAI key.
+KEYWORD_LLM_BINDING_API_KEY=local-vllm-api-key
+KEYWORD_OPENAI_LLM_MAX_TOKENS=2048
+# Optional for Qwen-style models served by vLLM when you want to disable thinking.
+KEYWORD_OPENAI_LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'
+KEYWORD_MAX_ASYNC_LLM=4
+KEYWORD_LLM_TIMEOUT=60
+```
+
+This pattern is not cross-provider because all three roles use the `openai` binding. LightRAG passes each role's `*_LLM_BINDING_HOST` and `*_LLM_BINDING_API_KEY` to the OpenAI-compatible client separately.
+
+Note: provider options within the same provider inherit the base `OPENAI_LLM_*`. If the local vLLM server does not support official OpenAI parameters such as `reasoning_effort`, do not set the global `OPENAI_LLM_REASONING_EFFORT`; use role-level variables such as `EXTRACT_OPENAI_LLM_REASONING_EFFORT` and `QUERY_OPENAI_LLM_REASONING_EFFORT` instead.
+
+### 4. One Role Crosses Provider
+
+Suitable when the base uses an official OpenAI model and only keyword extraction uses local Ollama:
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_openai_api_key
+OPENAI_LLM_REASONING_EFFORT=medium
+
+KEYWORD_LLM_BINDING=ollama
+KEYWORD_LLM_MODEL=qwen3.5:9b
+KEYWORD_LLM_BINDING_HOST=http://localhost:11434
+KEYWORD_LLM_BINDING_API_KEY=ollama-local-key
+KEYWORD_OLLAMA_LLM_NUM_CTX=32768
+```
+
+For cross-provider configurations, Ollama options do not inherit OpenAI options. For local Ollama, `KEYWORD_LLM_BINDING_API_KEY` can usually use a placeholder value; the current cross-provider validation requires non-Bedrock roles to explicitly provide a role-level API key.
+
+### 5. Specify a Dedicated Multimodal Model for VLM
+
+Suitable when text tasks use a cheaper model and multimodal analysis uses a vision-language model:
+
+```env
+VLM_PROCESS_ENABLE=true
+
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_api_key
+
+VLM_LLM_BINDING=openai
+VLM_LLM_MODEL=gpt-4o
+VLM_OPENAI_LLM_MAX_TOKENS=4096
+VLM_MAX_ASYNC_LLM=2
+VLM_LLM_TIMEOUT=240
+```
+
+If VLM uses the same provider and key, `VLM_LLM_BINDING_HOST` and `VLM_LLM_BINDING_API_KEY` can be omitted.
+
+`VLM_PROCESS_ENABLE` is the master switch for multimodal analysis. When `false`, the pipeline emits a warning and skips every multimodal item without invoking the VLM. When `true`, the effective VLM binding (`VLM_LLM_BINDING` if set, otherwise `LLM_BINDING`) must support image inputs. The following providers are vision-capable: `openai`, `azure_openai`, `gemini`, `bedrock`, `ollama`, `anthropic`. `lollms` is rejected at startup because it cannot accept image inputs.
+
+### 6. Bedrock Role-Level SigV4 Credentials
+
+Suitable when only one role accesses Bedrock and uses independent IAM/STS credentials:
+
+```env
+LLM_BINDING=openai
+LLM_MODEL=gpt-5-mini
+LLM_BINDING_HOST=https://api.openai.com/v1
+LLM_BINDING_API_KEY=your_openai_api_key
+
+QUERY_LLM_BINDING=bedrock
+QUERY_LLM_MODEL=us.amazon.nova-lite-v1:0
+QUERY_LLM_BINDING_HOST=DEFAULT_BEDROCK_ENDPOINT
+QUERY_AWS_REGION=us-east-1
+QUERY_AWS_ACCESS_KEY_ID=your_query_access_key
+QUERY_AWS_SECRET_ACCESS_KEY=your_query_secret_key
+QUERY_AWS_SESSION_TOKEN=your_optional_session_token
+QUERY_BEDROCK_LLM_MAX_TOKENS=4096
+QUERY_BEDROCK_LLM_TEMPERATURE=0.2
+```
+
+Do not set `QUERY_LLM_BINDING_API_KEY`; Bedrock rejects that configuration.
+
+## Caveats
+
+- Within the same provider, provider options such as `OPENAI_LLM_REASONING_EFFORT`, `OPENAI_LLM_MAX_TOKENS`, `OLLAMA_LLM_NUM_CTX`, and `GEMINI_LLM_THINKING_CONFIG` are inherited automatically.
+- There is currently no clean role-level semantic for "unsetting an inherited provider option". If a model in a same-provider role does not support a base option, explicitly override that option for the role with a supported value, or configure the role as cross-provider and set only the role-specific provider options it supports.
+- `AZURE_OPENAI_DEPLOYMENT` and `AZURE_OPENAI_API_VERSION` for `azure_openai` are global environment variables. If `AZURE_OPENAI_DEPLOYMENT` is set, it may take precedence over the role model name.
+- Gemini Vertex AI mode is controlled by process-level Google environment variables. In the same LightRAG process, some roles cannot use Vertex AI while others use AI Studio API keys.
+- In Docker/Compose, `LLM_BINDING_HOST` usually needs to use a container-reachable address such as `host.docker.internal`; role-level hosts follow the same principle.
+- Restart LightRAG Server after modifying `.env`. Some IDE terminals preload `.env`, so opening a new terminal session is recommended to confirm that environment variables take effect.

docs/ThirdPartyParser-zh.md

@@ -0,0 +1,214 @@
+# 第三方 Parser 引擎开发与注册指南
+
+LightRAG 的解析层通过统一的 `BaseParser` 契约 + 中央引擎注册表(`lightrag/parser/registry.py`)派发所有解析引擎。内置引擎(`native` / `legacy` / `mineru` / `docling`)与第三方引擎走完全相同的派发路径:pipeline worker 与调试 CLI 都通过 `get_parser(engine).parse(ParseContext(...))` 驱动,**没有任何针对内置引擎的特判**。因此第三方包只需做两件事:
+
+1. **实现**一个 `BaseParser` 子类;
+2. **注册**一个 `ParserSpec`(推荐通过 `lightrag.parsers` entry point 自动发现)。
+
+完成后,该引擎自动获得:独立(或共享)的解析并发池、文件名 hint / `LIGHTRAG_PARSER` 路由规则 / API `parse_engine` 参数三种选择方式、后缀能力校验、以及 `python -m lightrag.parser.cli --engine <name>` 单文件调试支持。
+
+> 架构背景见 RFC #3197;sidecar 文件格式见 `docs/LightRAGSidecarFormat-zh.md`;CLI 用法见 `docs/ParserDebugCLI-zh.md`。
+
+---
+
+## 1. 派发流程总览
+
+```
+上传/扫描 → enqueue(PENDING_PARSE, parse_engine=<engine>)
+    → pipeline 按 ParserSpec.queue_group 选并发池
+    → parse worker: get_parser(engine).parse(ParseContext(rag, doc_id, file_path, content_data))
+        ├─ 成功 → ParseResult → 进入 analyze / chunk / KG 流水线
+        └─ 抛异常 → 该文档 doc_status=FAILED(只影响这一篇)
+```
+
+引擎对单篇文档的全部职责都收敛在一次 `parse(ctx)` 调用里:解析、持久化 `full_docs`、归档源文件、返回结构化结果。
+
+## 2. 实现 Parser
+
+### 2.1 契约(`lightrag/parser/base.py`)
+
+```python
+class MyParser(BaseParser):
+    engine_name = "myengine"          # 必须与 ParserSpec.engine_name 一致
+
+    async def parse(self, ctx: ParseContext) -> ParseResult: ...
+```
+
+`ParseContext` 提供:
+
+| 成员 | 说明 |
+|---|---|
+| `ctx.rag` | LightRAG 实例(用于 `_persist_parsed_full_docs` 等) |
+| `ctx.doc_id` / `ctx.file_path` / `ctx.content_data` | 文档标识、规范化文件路径、`full_docs` 行 |
+| `ctx.resolve(engine_name)` | 返回 `ResolvedSource(source_path, document_name, parsed_dir)` —— 解析磁盘源文件路径、规范化文档名、推导 `__parsed__/<base>.parsed/` 产物目录 |
+| `ctx.archive_source(path)` | 解析成功并完成 `full_docs` 同步后,把源文件归档进 `__parsed__/` |
+
+`ParseResult` 字段:`doc_id` / `file_path` / `parse_format`(`"raw"` 或 `"lightrag"`)/ `content` / `blocks_path`(无 sidecar 则 `""`)/ `parse_engine` / `parse_stage_skipped`(缓存命中等跳过场景)/ `parse_warnings`(非致命警告,会落到 `doc_status.metadata`)。
+
+### 2.2 三条实现路径(按引擎形态选基类)
+
+**A. 纯文本引擎(无 sidecar)— 直接继承 `BaseParser`**
+
+参考 `lightrag/parser/legacy/parser.py`(`LegacyParser`),核心骨架:
+
+```python
+class MyTextParser(BaseParser):
+    engine_name = "myengine"
+
+    async def parse(self, ctx: ParseContext) -> ParseResult:
+        rs = ctx.resolve(self.engine_name)
+        source = rs.source_path
+        if not source.is_file():
+            raise FileNotFoundError(f"myengine source not found: {source}")
+
+        text = await asyncio.to_thread(my_extract, source)  # CPU 活进线程
+        if not text.strip():
+            raise ValueError(f"extracted no usable text from {ctx.file_path}")
+
+        await ctx.rag._persist_parsed_full_docs(ctx.doc_id, {
+            "content": text,
+            "file_path": ctx.file_path,
+            "parse_format": FULL_DOCS_FORMAT_RAW,
+            "parse_engine": self.engine_name,
+            "update_time": int(time.time()),
+        })
+        await ctx.archive_source(str(source))
+        return ParseResult(
+            doc_id=ctx.doc_id, file_path=ctx.file_path,
+            parse_format=FULL_DOCS_FORMAT_RAW, content=text,
+            blocks_path="", parse_engine=self.engine_name,
+        )
+```
+
+**B. 本地解析、产 sidecar — 继承 `NativeParserBase`**(`lightrag/parser/native_base.py`)
+
+模板固定了「预清理产物目录(带回滚)→ 线程内抽取 → 构建 IR → 写 sidecar → 持久化 → 归档」的完整流程,只需实现两个钩子:
+
+```python
+class MyNativeParser(NativeParserBase):
+    engine_name = "myengine"
+
+    def extract(self, source, *, parsed_dir, asset_dir, base_name):
+        """同步,在线程中运行;返回 (blocks, warnings, metadata)。
+        可在 write_sidecar 之前把图片等资产写入 asset_dir。"""
+
+    def build_ir(self, blocks, *, document_name, asset_dir_name, metadata) -> IRDoc:
+        """blocks → IRDoc(交给统一的 sidecar writer)。"""
+```
+
+可选覆写:`validate_source`(默认仅要求文件存在)、`surface_warnings`(把抽取警告映射为 `parse_warnings`)。参考实现:`lightrag/parser/docx/parser.py`。
+
+**C. 外部解析服务(下载 raw bundle + 缓存)— 继承 `ExternalParserBase`**(`lightrag/parser/external/_base.py`)
+
+模板固定了「raw 缓存命中检查 → 未命中则清目录重新下载 → 构建 IR → 写 sidecar → 持久化 → 归档」,实现三个钩子 + 两个类属性:
+
+```python
+class MyExternalParser(ExternalParserBase):
+    engine_name = "myengine"
+    raw_dir_suffix = ".myengine_raw"          # raw bundle 目录后缀(以 . 开头)
+    force_reparse_env = "LIGHTRAG_FORCE_REPARSE_MYENGINE"
+
+    def is_bundle_valid(self, raw_dir, source_path) -> bool: ...   # 缓存命中检查
+    async def download_into(self, raw_dir, source_path, *, upload_name): ...
+    def build_ir(self, raw_dir, document_name) -> IRDoc: ...
+```
+
+可选覆写 `validate_ir`(构建后校验,如零 block 报错)。参考实现:`lightrag/parser/external/mineru/parser.py`、`.../docling/parser.py`。
+
+### 2.3 失败语义(重要)
+
+- `parse(ctx)` **抛出任何异常 ⇒ 仅该文档标记 FAILED**,错误信息写入 `doc_status.error_msg`,不影响批内其他文档。
+- 解析出**空内容时应抛异常**而不是返回空串——否则会产出一篇零知识文档静默进入 chunking(内置引擎均遵循此约定)。
+- worker 在调用引擎前会做后缀守门:`PENDING_PARSE` 文档的后缀不在该引擎 `ParserSpec.suffixes` 内 ⇒ 直接 FAILED,引擎代码不会被调用。
+
+## 3. 声明 `ParserSpec`(能力元数据)
+
+```python
+from lightrag.parser.registry import ParserSpec, register_parser
+
+register_parser(ParserSpec(
+    engine_name="myengine",
+    impl="my_pkg.parser:MyParser",       # "module:Class",get_parser 时才懒加载
+    suffixes=frozenset({"pdf", "foo"}),  # 小写、不带点
+    queue_group="myengine",              # 见下文并发模型
+    concurrency=int(os.getenv("MAX_PARALLEL_PARSE_MYENGINE", "2")),
+    # 仅外部服务引擎需要(routing 在 endpoint 未配置时跳过该引擎):
+    endpoint_configured=lambda: bool(os.getenv("MYENGINE_ENDPOINT", "").strip()),
+    endpoint_requirement=lambda: "MYENGINE_ENDPOINT",
+))
+```
+
+| 字段 | 必填 | 说明 |
+|---|---|---|
+| `engine_name` | ✓ | 注册表键,也是 `--engine` / 文件名 hint / `LIGHTRAG_PARSER` 里的引擎名。**与已有名字相同会覆盖原注册**(包括内置引擎)——除非有意替换实现,请勿与 `native/legacy/mineru/docling` 撞名。 |
+| `impl` | ✓ | `"module:Class"` 字符串。注册表只在文档实际解析时才 import 它,**注册阶段绝不能提前 import 实现**(保持能力查询 import-cheap,这是注册表的设计不变量)。 |
+| `suffixes` | ✓ | 该引擎能处理的扩展名(小写无点)。用于路由校验与 worker 端后缀守门。 |
+| `queue_group` | | 并发池分组,默认 `"native"`(共享 native 池)。独立池填唯一组名。 |
+| `concurrency` | | 该组 worker 数(组的唯一 owner 才需要填)。环境变量覆盖由**注册方在注册时自行烘焙**(如上例 `int(os.getenv(...))`),注册值即权威值。 |
+| `endpoint_configured` / `endpoint_requirement` | | 零参闭包(只读 env、不发网络)。前者返回该引擎依赖的外部服务是否已配置;后者返回缺失时提示用户的配置项名。本地引擎不用填(默认恒可用)。 |
+| `user_selectable` | | 默认 `True`。`False` 表示内部格式 handler(如 `reuse`/`passthrough`),不会出现在引擎选择面。 |
+
+### 并发模型
+
+- pipeline 每批为**每个 `queue_group` 建一条队列 + 一组 worker**;
+- 组的 worker 数:内置组(`native`/`mineru`/`docling`)由 LightRAG 实例字段 `max_parallel_parse_*` 决定(支持构造参数覆盖);第三方独立组取该组唯一 owner spec 的 `concurrency`;一个组**只能有一个**声明了 `concurrency` 的 spec,多个会在批启动时报错;
+- `queue_group="native"` 蹭内置池时,`concurrency` 不生效(池大小由 `max_parallel_parse_native` 决定,被忽略的 spec 级 `concurrency` 会在批启动时记录 warning 日志)——本地轻量引擎(如 legacy)适合这种方式,外部服务引擎建议独立组以免慢请求拖住本地解析。
+
+## 4. 注册:entry point 自动发现(推荐)
+
+LightRAG 通过 `lightrag.parsers` entry-point group 自动发现第三方引擎(`lightrag/parser/plugins.py`)。第三方包只需:
+
+**① 在自己的 `pyproject.toml` 声明 entry point:**
+
+```toml
+[project.entry-points."lightrag.parsers"]
+myengine = "my_pkg.lightrag_plugin:register"
+```
+
+**② 提供一个零参注册函数:**
+
+```python
+# my_pkg/lightrag_plugin.py —— 保持 import-cheap:不要在这里 import 解析实现
+import os
+from lightrag.parser.registry import ParserSpec, register_parser
+
+def register() -> None:
+    register_parser(ParserSpec(
+        engine_name="myengine",
+        impl="my_pkg.parser:MyParser",   # 实现走懒加载
+        suffixes=frozenset({"foo"}),
+        queue_group="myengine",
+        concurrency=int(os.getenv("MAX_PARALLEL_PARSE_MYENGINE", "2")),
+    ))
+```
+
+`pip install my-pkg` 之后即装即用,无需改动 LightRAG 代码:
+
+- **API Server**:`create_app()` 在校验 `LIGHTRAG_PARSER` 路由规则**之前**调用 `load_third_party_parsers()`,因此路由规则可以直接引用第三方引擎名(如 `LIGHTRAG_PARSER="foo:myengine"`)。上传与扫描的文件类型守门**完全由注册表 + 路由实时派生**,判据是"这份文件能否路由到一个支持它的引擎":裸后缀(无 hint)要求 `LIGHTRAG_PARSER` 规则把它指到你的引擎(否则默认 legacy 接不住会被拒,而不是收下后在解析阶段 FAILED);带文件名 hint 的上传(如 `report.[myengine].foo`)无需规则即可通过;endpoint 未配置的引擎其独有后缀不参与(与内置 mineru/docling 的图片格式同一规则)。**实践建议:发布第三方引擎时,在部署说明里让用户配套 `LIGHTRAG_PARSER="foo:myengine"` 规则**,裸文件名上传与目录扫描即自动生效;
+- **调试 CLI**:`python -m lightrag.parser.cli sample.foo --engine myengine` 直接可用(`main()` 在构建 `--engine` choices 前加载插件)。无 sidecar 的引擎(`blocks_path=""`)CLI 会打印纯文本摘要而非 blocks 摘要;继承 `ExternalParserBase` 的引擎自动获得 raw 缓存展示与 `--force-reparse` 支持;
+- **库内嵌用法**(不经 server/CLI 直接用 LightRAG 类):在构建 pipeline 前自行调用一次:
+
+```python
+from lightrag.parser.plugins import load_third_party_parsers
+load_third_party_parsers()   # 进程内幂等
+```
+
+加载语义:每进程幂等(重复调用为 no-op);**单个插件抛异常只会被记录并跳过**,不会影响其他插件或内置引擎,更不会阻断 server 启动——但该引擎将不可用,请关注启动日志中的 `[parser-plugins]` 行。
+
+> 不想发包时也可以跳过 entry point,在自己的启动脚本里直接 `register_parser(...)` 后再启动/调用 LightRAG——注册表是进程内模块单例,效果相同,只是没有"装上即生效"。
+
+## 5. 路由:让文档用上你的引擎
+
+引擎选择优先级(`lightrag/parser/routing.py`):
+
+1. **文件名 hint**:`report.[myengine].foo`(可带处理选项 `report.[myengine-iet].foo`);
+2. **`LIGHTRAG_PARSER` 规则**:如 `LIGHTRAG_PARSER="foo:myengine,pdf:mineru"`(按后缀 glob 匹配,首条命中生效);
+3. **默认**:`legacy`。
+
+API 上传时显式传 `parse_engine="myengine"` 则直接固定该引擎(存入 PENDING_PARSE 行,worker 原样履约;后缀不支持会 FAILED 而非静默回退)。注册了 `endpoint_configured` 的引擎在 endpoint 未配置时会被路由跳过(hint/规则校验也会给出 `endpoint_requirement` 提示)。
+
+## 6. 测试建议
+
+- **单测引擎本体**:绕开 CLI,直接 `get_parser("myengine").parse(ParseContext(fake_rag, doc_id, file_path, content_data))`。`fake_rag` 只需提供 `_persist_parsed_full_docs` / `_resolve_source_file_for_parser` / `full_docs` / `doc_status`(参考 `lightrag/parser/debug.py` 的 `build_debug_rag()`,或 `tests/parser/test_legacy_parser.py` 的极简 `_FakeRag`);
+- **注册表是模块级单例**:测试中 `register_parser` 后,用 `finally: registry._REGISTRY.pop("myengine", None)` 清理(参考 `tests/parser/test_registry.py`);
+- **entry-point 加载逻辑**:参考 `tests/parser/test_plugins.py`(monkeypatch `lightrag.parser.plugins.entry_points` 注入 fake entry point;`plugins._loaded` 标志需复位)。

docs/ThirdPartyParser.md

@@ -0,0 +1,214 @@
+# Third-Party Parser Engine Development and Registration Guide
+
+LightRAG's parsing layer dispatches all parsing engines through a unified `BaseParser` contract plus the central engine registry (`lightrag/parser/registry.py`). Built-in engines (`native` / `legacy` / `mineru` / `docling`) and third-party engines use the exact same dispatch path: both pipeline workers and the debug CLI drive parsing through `get_parser(engine).parse(ParseContext(...))`, with **no special cases for built-in engines**. Therefore, a third-party package only needs to do two things:
+
+1. **Implement** a `BaseParser` subclass;
+2. **Register** a `ParserSpec` (automatic discovery through the `lightrag.parsers` entry point is recommended).
+
+Once that is done, the engine automatically gets: a dedicated (or shared) parsing concurrency pool, three engine-selection methods (filename hints / `LIGHTRAG_PARSER` routing rules / API `parse_engine` parameter), suffix capability validation, and single-file debug support through `python -m lightrag.parser.cli --engine <name>`.
+
+> For architecture background, see RFC #3197; for the sidecar file format, see `docs/LightRAGSidecarFormat.md`; for CLI usage, see `docs/ParserDebugCLI.md`.
+
+---
+
+## 1. Dispatch Flow Overview
+
+```
+Upload/scan -> enqueue(PENDING_PARSE, parse_engine=<engine>)
+    -> pipeline selects a concurrency pool by ParserSpec.queue_group
+    -> parse worker: get_parser(engine).parse(ParseContext(rag, doc_id, file_path, content_data))
+        +- success -> ParseResult -> enter analyze / chunk / KG pipeline
+        +- exception -> this document's doc_status=FAILED (only this document is affected)
+```
+
+For a single document, all engine responsibilities converge into one `parse(ctx)` call: parsing, persisting `full_docs`, archiving the source file, and returning a structured result.
+
+## 2. Implementing a Parser
+
+### 2.1 Contract (`lightrag/parser/base.py`)
+
+```python
+class MyParser(BaseParser):
+    engine_name = "myengine"          # Must match ParserSpec.engine_name
+
+    async def parse(self, ctx: ParseContext) -> ParseResult: ...
+```
+
+`ParseContext` provides:
+
+| Member | Description |
+|---|---|
+| `ctx.rag` | LightRAG instance (used for `_persist_parsed_full_docs`, etc.) |
+| `ctx.doc_id` / `ctx.file_path` / `ctx.content_data` | Document identifier, normalized file path, and `full_docs` row |
+| `ctx.resolve(engine_name)` | Returns `ResolvedSource(source_path, document_name, parsed_dir)`: resolves the on-disk source file path, normalized document name, and derived `__parsed__/<base>.parsed/` artifact directory |
+| `ctx.archive_source(path)` | After parsing succeeds and `full_docs` synchronization is complete, archives the source file into `__parsed__/` |
+
+`ParseResult` fields: `doc_id` / `file_path` / `parse_format` (`"raw"` or `"lightrag"`) / `content` / `blocks_path` (`""` when there is no sidecar) / `parse_engine` / `parse_stage_skipped` (skipped scenarios such as cache hits) / `parse_warnings` (non-fatal warnings, persisted to `doc_status.metadata`).
+
+### 2.2 Three Implementation Paths (Choose a Base Class by Engine Type)
+
+**A. Plain-text engine (no sidecar) - inherit `BaseParser` directly**
+
+See `lightrag/parser/legacy/parser.py` (`LegacyParser`). Core skeleton:
+
+```python
+class MyTextParser(BaseParser):
+    engine_name = "myengine"
+
+    async def parse(self, ctx: ParseContext) -> ParseResult:
+        rs = ctx.resolve(self.engine_name)
+        source = rs.source_path
+        if not source.is_file():
+            raise FileNotFoundError(f"myengine source not found: {source}")
+
+        text = await asyncio.to_thread(my_extract, source)  # Run CPU work in a thread
+        if not text.strip():
+            raise ValueError(f"extracted no usable text from {ctx.file_path}")
+
+        await ctx.rag._persist_parsed_full_docs(ctx.doc_id, {
+            "content": text,
+            "file_path": ctx.file_path,
+            "parse_format": FULL_DOCS_FORMAT_RAW,
+            "parse_engine": self.engine_name,
+            "update_time": int(time.time()),
+        })
+        await ctx.archive_source(str(source))
+        return ParseResult(
+            doc_id=ctx.doc_id, file_path=ctx.file_path,
+            parse_format=FULL_DOCS_FORMAT_RAW, content=text,
+            blocks_path="", parse_engine=self.engine_name,
+        )
+```
+
+**B. Local parser that produces a sidecar - inherit `NativeParserBase`** (`lightrag/parser/native_base.py`)
+
+The template fixes the complete flow of "pre-clean artifact directory (with rollback) -> extract in a thread -> build IR -> write sidecar -> persist -> archive". You only need to implement two hooks:
+
+```python
+class MyNativeParser(NativeParserBase):
+    engine_name = "myengine"
+
+    def extract(self, source, *, parsed_dir, asset_dir, base_name):
+        """Synchronous, runs in a thread; returns (blocks, warnings, metadata).
+        Assets such as images can be written to asset_dir before write_sidecar."""
+
+    def build_ir(self, blocks, *, document_name, asset_dir_name, metadata) -> IRDoc:
+        """blocks -> IRDoc (handed to the shared sidecar writer)."""
+```
+
+Optional overrides: `validate_source` (by default only requires the file to exist), `surface_warnings` (maps extraction warnings to `parse_warnings`). Reference implementation: `lightrag/parser/docx/parser.py`.
+
+**C. External parsing service (download raw bundle + cache) - inherit `ExternalParserBase`** (`lightrag/parser/external/_base.py`)
+
+The template fixes the flow of "raw cache-hit check -> if missed, clear the directory and download again -> build IR -> write sidecar -> persist -> archive". Implement three hooks plus two class attributes:
+
+```python
+class MyExternalParser(ExternalParserBase):
+    engine_name = "myengine"
+    raw_dir_suffix = ".myengine_raw"          # Raw bundle directory suffix (starts with .)
+    force_reparse_env = "LIGHTRAG_FORCE_REPARSE_MYENGINE"
+
+    def is_bundle_valid(self, raw_dir, source_path) -> bool: ...   # Cache-hit check
+    async def download_into(self, raw_dir, source_path, *, upload_name): ...
+    def build_ir(self, raw_dir, document_name) -> IRDoc: ...
+```
+
+Optional override: `validate_ir` (post-build validation, for example failing on zero blocks). Reference implementations: `lightrag/parser/external/mineru/parser.py`, `.../docling/parser.py`.
+
+### 2.3 Failure Semantics (Important)
+
+- If `parse(ctx)` **raises any exception, only that document is marked FAILED**. The error message is written to `doc_status.error_msg`, and other documents in the batch are not affected.
+- If parsing produces **empty content, raise an exception** instead of returning an empty string. Otherwise, a zero-knowledge document would silently enter chunking (all built-in engines follow this convention).
+- Before calling the engine, the worker performs suffix guarding: if a `PENDING_PARSE` document's suffix is not included in that engine's `ParserSpec.suffixes`, the document is FAILED directly and the engine code is never called.
+
+## 3. Declaring `ParserSpec` (Capability Metadata)
+
+```python
+from lightrag.parser.registry import ParserSpec, register_parser
+
+register_parser(ParserSpec(
+    engine_name="myengine",
+    impl="my_pkg.parser:MyParser",       # "module:Class", lazy-loaded by get_parser
+    suffixes=frozenset({"pdf", "foo"}),  # Lowercase, no dot
+    queue_group="myengine",              # See concurrency model below
+    concurrency=int(os.getenv("MAX_PARALLEL_PARSE_MYENGINE", "2")),
+    # Required only by external-service engines (routing skips this engine when no endpoint is configured):
+    endpoint_configured=lambda: bool(os.getenv("MYENGINE_ENDPOINT", "").strip()),
+    endpoint_requirement=lambda: "MYENGINE_ENDPOINT",
+))
+```
+
+| Field | Required | Description |
+|---|---|---|
+| `engine_name` | yes | Registry key; also the engine name used by `--engine`, filename hints, and `LIGHTRAG_PARSER`. **Registering the same name as an existing engine overwrites the previous registration** (including built-in engines). Avoid colliding with `native/legacy/mineru/docling` unless you intentionally want to replace that implementation. |
+| `impl` | yes | `"module:Class"` string. The registry imports it only when a document is actually parsed. **The implementation must never be imported early during registration** (capability queries must stay import-cheap; this is a registry design invariant). |
+| `suffixes` | yes | File extensions the engine can handle (lowercase, no dot). Used for routing validation and worker-side suffix guarding. |
+| `queue_group` | | Concurrency-pool group. Defaults to `"native"` (sharing the native pool). Use a unique group name for a dedicated pool. |
+| `concurrency` | | Number of workers for this group (only needed for the group's sole owner). Environment-variable overrides are **baked by the registration code at registration time** (as in the `int(os.getenv(...))` example above); the registered value is authoritative. |
+| `endpoint_configured` / `endpoint_requirement` | | Zero-argument closures (read env only, no network calls). The former returns whether the external service required by this engine is configured; the latter returns the configuration item name to show users when it is missing. Local engines do not need these fields (available by default). |
+| `user_selectable` | | Defaults to `True`. `False` means an internal format handler (such as `reuse` / `passthrough`) that is not shown as a selectable engine. |
+
+### Concurrency Model
+
+- For each batch, the pipeline creates **one queue plus one worker group for every `queue_group`**;
+- Worker count for a group: built-in groups (`native` / `mineru` / `docling`) are determined by LightRAG instance fields `max_parallel_parse_*` (constructor overrides are supported); a third-party dedicated group uses the `concurrency` value from the group's sole owner spec; **only one** spec in a group may declare `concurrency`, otherwise batch startup fails;
+- When using `queue_group="native"` to share the built-in pool, `concurrency` does not take effect (pool size is determined by `max_parallel_parse_native`; ignored spec-level `concurrency` values are recorded as warning logs at batch startup). Lightweight local engines (such as `legacy`) fit this mode, while external-service engines should usually use a dedicated group so slow requests do not block local parsing.
+
+## 4. Registration: Automatic Discovery via Entry Point (Recommended)
+
+LightRAG automatically discovers third-party engines through the `lightrag.parsers` entry-point group (`lightrag/parser/plugins.py`). A third-party package only needs to:
+
+**1. Declare the entry point in its own `pyproject.toml`:**
+
+```toml
+[project.entry-points."lightrag.parsers"]
+myengine = "my_pkg.lightrag_plugin:register"
+```
+
+**2. Provide a zero-argument registration function:**
+
+```python
+# my_pkg/lightrag_plugin.py - keep imports cheap: do not import parser implementations here
+import os
+from lightrag.parser.registry import ParserSpec, register_parser
+
+def register() -> None:
+    register_parser(ParserSpec(
+        engine_name="myengine",
+        impl="my_pkg.parser:MyParser",   # Implementation is lazy-loaded
+        suffixes=frozenset({"foo"}),
+        queue_group="myengine",
+        concurrency=int(os.getenv("MAX_PARALLEL_PARSE_MYENGINE", "2")),
+    ))
+```
+
+After `pip install my-pkg`, it works immediately without changing LightRAG code:
+
+- **API Server**: `create_app()` calls `load_third_party_parsers()` **before** validating `LIGHTRAG_PARSER` routing rules, so routing rules can directly reference third-party engine names (for example, `LIGHTRAG_PARSER="foo:myengine"`). File-type guarding for uploads and scans is **derived entirely from the registry plus routing at runtime**. The criterion is "can this file route to an engine that supports it": a bare suffix (no hint) requires a `LIGHTRAG_PARSER` rule that routes it to your engine (otherwise the default `legacy` engine cannot handle it, so the file is rejected instead of being accepted and later FAILED during parsing); uploads with a filename hint (for example, `report.[myengine].foo`) pass without a rule; unique suffixes of engines whose endpoints are not configured do not participate (the same rule used by built-in mineru/docling image formats). **Practical recommendation: when publishing a third-party engine, tell users to configure the matching `LIGHTRAG_PARSER="foo:myengine"` rule in deployment docs**, so bare-filename uploads and directory scans work automatically;
+- **Debug CLI**: `python -m lightrag.parser.cli sample.foo --engine myengine` works directly (`main()` loads plugins before building `--engine` choices). For engines without sidecars (`blocks_path=""`), the CLI prints a plain-text summary instead of a block summary; engines inheriting `ExternalParserBase` automatically get raw cache display and `--force-reparse` support;
+- **Embedded library usage** (using the `LightRAG` class directly without going through the server or CLI): call this once before building the pipeline:
+
+```python
+from lightrag.parser.plugins import load_third_party_parsers
+load_third_party_parsers()   # Idempotent within the process
+```
+
+Loading semantics: idempotent per process (repeated calls are no-ops); **if a single plugin raises an exception, it is only logged and skipped**. It does not affect other plugins or built-in engines, and it does not block server startup. However, that engine will be unavailable, so watch for `[parser-plugins]` lines in startup logs.
+
+> If you do not want to publish a package, you can skip entry points and directly call `register_parser(...)` in your own startup script before starting/calling LightRAG. The registry is an in-process module singleton, so the effect is the same, just without "install and it works" behavior.
+
+## 5. Routing: Making Documents Use Your Engine
+
+Engine selection priority (`lightrag/parser/routing.py`):
+
+1. **Filename hint**: `report.[myengine].foo` (processing options are allowed, such as `report.[myengine-iet].foo`);
+2. **`LIGHTRAG_PARSER` rules**: for example, `LIGHTRAG_PARSER="foo:myengine,pdf:mineru"` (matched by suffix glob; the first match wins);
+3. **Default**: `legacy`.
+
+When API upload explicitly passes `parse_engine="myengine"`, that engine is fixed directly (stored in the `PENDING_PARSE` row and honored verbatim by the worker; unsupported suffixes are FAILED instead of silently falling back). Engines that register `endpoint_configured` are skipped by routing when the endpoint is not configured (hint/rule validation also shows the `endpoint_requirement` prompt).
+
+## 6. Testing Recommendations
+
+- **Unit-test the engine itself**: bypass the CLI and directly call `get_parser("myengine").parse(ParseContext(fake_rag, doc_id, file_path, content_data))`. `fake_rag` only needs to provide `_persist_parsed_full_docs` / `_resolve_source_file_for_parser` / `full_docs` / `doc_status` (see `build_debug_rag()` in `lightrag/parser/debug.py`, or the minimal `_FakeRag` in `tests/parser/test_legacy_parser.py`);
+- **The registry is a module-level singleton**: after calling `register_parser` in a test, clean up with `finally: registry._REGISTRY.pop("myengine", None)` (see `tests/parser/test_registry.py`);
+- **Entry-point loading logic**: see `tests/parser/test_plugins.py` (monkeypatch `lightrag.parser.plugins.entry_points` to inject a fake entry point; the `plugins._loaded` flag must be reset).

docs/UV_LOCK_GUIDE.md

@@ -0,0 +1,170 @@
+# uv.lock Update Guide
+
+## What is uv.lock?
+
+`uv.lock` is uv's lock file. It captures the exact version of every dependency, including transitive ones, much like:
+- Node.js `package-lock.json`
+- Rust `Cargo.lock`
+- Python Poetry `poetry.lock`
+
+Keeping `uv.lock` in version control guarantees that everyone installs the same dependency set.
+
+## When does uv.lock change?
+
+### Situations where it does *not* change automatically
+
+- Running `uv sync --frozen`
+- Building Docker images that call `uv sync --frozen`
+- Editing source code without touching dependency metadata
+
+### Situations where it will change
+
+1. **`uv lock` or `uv lock --upgrade`**
+
+   ```bash
+   uv lock                # Resolve according to current constraints
+   uv lock --upgrade      # Re-resolve and upgrade to the newest compatible releases
+   ```
+
+   Use these commands after modifying `pyproject.toml`, when you want fresh dependency versions, or if the lock file was deleted or corrupted.
+
+2. **`uv add`**
+
+   ```bash
+    uv add requests           # Adds the dependency and updates both files
+    uv add --dev pytest       # Adds a dev dependency
+   ```
+
+   `uv add` edits `pyproject.toml` and refreshes `uv.lock` in one step.
+
+3. **`uv remove`**
+
+   ```bash
+   uv remove requests
+   ```
+
+   This removes the dependency from `pyproject.toml` and rewrites `uv.lock`.
+
+4. **`uv sync` without `--frozen`**
+
+   ```bash
+   uv sync
+   ```
+
+   Normally this only installs what is already locked. However, if `pyproject.toml` and `uv.lock` disagree or the lock file is missing, uv will regenerate and update `uv.lock`. In CI and production builds you should prefer `uv sync --frozen` to prevent unintended updates.
+
+## Example workflows
+
+### Scenario 1: Add a new dependency
+
+```bash
+# Recommended: let uv handle both files
+uv add fastapi
+git add pyproject.toml uv.lock
+git commit -m "Add fastapi dependency"
+
+# Manual alternative
+# 1. Edit pyproject.toml
+# 2. Regenerate the lock file
+uv lock
+git add pyproject.toml uv.lock
+git commit -m "Add fastapi dependency"
+```
+
+### Scenario 2: Relax or tighten a version constraint
+
+```bash
+# 1. Edit the requirement in pyproject.toml,
+#    e.g. openai>=1.0.0,<2.0.0 -> openai>=1.5.0,<2.0.0
+
+# 2. Re-resolve the lock file
+uv lock
+
+# 3. Commit both files
+git add pyproject.toml uv.lock
+git commit -m "Update openai to >=1.5.0"
+```
+
+### Scenario 3: Upgrade everything to the newest compatible versions
+
+```bash
+uv lock --upgrade
+git diff uv.lock
+git add uv.lock
+git commit -m "Upgrade dependencies to latest compatible versions"
+```
+
+### Scenario 4: Teammate syncing the project
+
+```bash
+git pull               # Fetch latest code and lock file
+uv sync --frozen       # Install exactly what uv.lock specifies
+```
+
+## Using uv.lock in Docker
+
+```dockerfile
+RUN uv sync --frozen --no-dev --extra api
+```
+
+`--frozen` guarantees reproducible builds because uv will refuse to deviate from the locked versions.
+`--extra api` install API server
+
+## Generating a lock file that includes offline dependencies
+
+If you need `uv.lock` to capture the optional offline stacks, regenerate it with the relevant extras enabled:
+
+```bash
+uv lock --extra api --extra offline
+```
+
+This command resolves the base project requirements plus both the `api` and `offline` optional dependency sets, ensuring downstream `uv sync --frozen --extra api --extra offline` installs work without further resolution.
+
+## Frequently asked questions
+
+- **`uv.lock` is almost 1 MB. Does that matter?**
+  No. The file is read only during dependency resolution.
+
+- **Should we commit `uv.lock`?**
+  Yes. Commit it so collaborators and CI jobs share the same dependency graph.
+
+- **Deleted the lock file by accident?**
+  Run `uv lock` to regenerate it from `pyproject.toml`.
+
+- **Can `uv.lock` and `requirements.txt` coexist?**
+  They can, but maintaining both is redundant. Prefer relying on `uv.lock` alone whenever possible.
+
+- **How do I inspect locked versions?**
+  ```bash
+  uv tree
+  grep -A5 'name = "openai"' uv.lock
+  ```
+
+## Best practices
+
+### Recommended
+
+1. Commit `uv.lock` alongside `pyproject.toml`.
+2. Use `uv sync --frozen` in CI, Docker, and other reproducible environments.
+3. Use plain `uv sync` during local development if you want uv to reconcile the lock for you.
+4. Run `uv lock --upgrade` periodically to pick up the latest compatible releases.
+5. Regenerate the lock file immediately after changing dependency constraints.
+
+### Avoid
+
+1. Running `uv sync` without `--frozen` in CI or production pipelines.
+2. Editing `uv.lock` by hand—uv will overwrite manual edits.
+3. Ignoring lock file diffs in code reviews—unexpected dependency changes can break builds.
+
+## Summary
+
+| Command               | Updates `uv.lock` | Typical use                               |
+|-----------------------|-------------------|-------------------------------------------|
+| `uv lock`             | ✅ Yes            | After editing constraints                 |
+| `uv lock --upgrade`   | ✅ Yes            | Upgrade to the newest compatible versions |
+| `uv add <pkg>`        | ✅ Yes            | Add a dependency                          |
+| `uv remove <pkg>`     | ✅ Yes            | Remove a dependency                       |
+| `uv sync`             | ⚠️ Maybe          | Local development; can regenerate the lock |
+| `uv sync --frozen`    | ❌ No             | CI/CD, Docker, reproducible builds        |
+
+Remember: `uv.lock` only changes when you run a command that tells it to. Keep it in sync with your project and commit it whenever it changes.