chore: import upstream snapshot with attribution

docs/CHANGELOG.md

@@ -0,0 +1,42 @@
+# Changelog
+
+Append-only log of major changes to LEANN (new features, breaking changes, important
+fixes). Newest entries at the bottom.
+
+## 2026-03-05: IVF backend incremental update support
+
+- Added `leann-backend-ivf` with FAISS IndexIVFFlat + DirectMap.Hashtable.
+- IVF supports in-place `add_vectors` and `remove_ids` without full rebuild.
+- `leann build` is now idempotent: re-running on an existing index does incremental update (add new, remove deleted, re-index modified files).
+- Fixed incremental build chunking inconsistency and shared metadata dict bug.
+- Fixed IVF incremental update duplicate chunks from stale `passages.jsonl`.
+
+## 2026-03-05: MCP server v2 — build, status, and structured search
+
+- Added `leann_build` MCP tool: build or incrementally update indexes directly from Claude Code.
+- Added `leann_status` MCP tool: inspect index details (backend, embedding model, chunk/file count, size).
+- `leann_search` now uses `--json` output with file paths always included, formatted as markdown code blocks.
+- Fixed `float32` JSON serialization bug in `leann search --json`.
+- Cleaned up MCP tool descriptions (concise, no emoji).
+
+## 2026-03-05: Documentation — roadmap, vision, and dev guidelines
+
+- Rewrote `docs/roadmap.md` with current P0/P1 priorities from GitHub issue #237.
+- Added `docs/ultimate_goal.md` — long-term vision (personal data platform, best code retrieval MCP, multimodal, local-first).
+- Added self-contained documentation principle and dev doc maintenance rules to `CLAUDE.md`.
+
+## 2026-06-02: GPU FlashLib IVF backend (`flashlib_ivf`)
+
+- Add `leann-backend-flashlib-ivf`, a GPU IVF-Flat (inverted file) approximate-NN
+  backend built on FlashLib (`flash_ivf_flat`, Triton/CuteDSL) — the GPU counterpart
+  of the FAISS `ivf` backend. Registered as backend name `flashlib_ivf`; install via
+  `uv sync --extra flashlib-ivf` or `pip install leann-backend-flashlib-ivf`. Shares
+  the `nlist`/`nprobe` recall knobs with the `ivf` backend, so the two are drop-in
+  comparable. Requires a CUDA GPU at build (k-means) and search.
+- Add `benchmarks/flashlib_ivf_vs_faiss_ivf.py`: head-to-head `flashlib_ivf` (GPU) vs
+  `ivf` (FAISS, CPU) at matched `nlist` across an `nprobe` sweep (build time,
+  single-query latency, batched throughput, recall@k vs exact ground truth). On an
+  NVIDIA H200 at 1M x 768 vectors (nlist=4096, 8 CPU threads): ~13x faster build and,
+  at nprobe=32, ~6.5x lower single-query latency / ~75x higher batched throughput at
+  comparable recall (GPU latency stays ~flat while CPU grows linearly with nprobe).
+- Docs: `docs/flashlib_backend_guide.md` gains a `flashlib_ivf` section.

docs/COLQWEN_GUIDE.md

@@ -0,0 +1,200 @@
+# ColQwen Integration Guide
+
+Easy-to-use multimodal PDF retrieval with ColQwen2/ColPali models.
+
+## Quick Start
+
+> **🍎 Mac Users**: ColQwen is optimized for Apple Silicon with MPS acceleration for faster inference!
+
+### 1. Install Dependencies
+```bash
+uv pip install colpali_engine pdf2image pillow matplotlib qwen_vl_utils einops seaborn
+brew install poppler  # macOS only, for PDF processing
+```
+
+### 2. Basic Usage
+```bash
+# Build index from PDFs
+python -m apps.colqwen_rag build --pdfs ./my_papers/ --index research_papers
+
+# Search with text queries
+python -m apps.colqwen_rag search research_papers "How does attention mechanism work?"
+
+# Interactive Q&A
+python -m apps.colqwen_rag ask research_papers --interactive
+```
+
+## Commands
+
+### Build Index
+```bash
+python -m apps.colqwen_rag build \
+  --pdfs ./pdf_directory/ \
+  --index my_index \
+  --model colqwen2 \
+  --pages-dir ./page_images/  # Optional: save page images
+```
+
+**Options:**
+- `--pdfs`: Directory containing PDF files (or single PDF path)
+- `--index`: Name for the index (required)
+- `--model`: `colqwen2` (default) or `colpali`
+- `--pages-dir`: Directory to save page images (optional)
+
+### Search Index
+```bash
+python -m apps.colqwen_rag search my_index "your question here" --top-k 5
+```
+
+**Options:**
+- `--top-k`: Number of results to return (default: 5)
+- `--model`: Model used for search (should match build model)
+
+### Interactive Q&A
+```bash
+python -m apps.colqwen_rag ask my_index --interactive
+```
+
+**Commands in interactive mode:**
+- Type your questions naturally
+- `help`: Show available commands
+- `quit`/`exit`/`q`: Exit interactive mode
+
+## 🧪 Test & Reproduce Results
+
+Run the reproduction test for issue #119:
+```bash
+python test_colqwen_reproduction.py
+```
+
+This will:
+1. ✅ Check dependencies
+2. 📥 Download sample PDF (Attention Is All You Need paper)
+3. 🏗️ Build test index
+4. 🔍 Run sample queries
+5. 📊 Show how to generate similarity maps
+
+## 🎨 Advanced: Similarity Maps
+
+For visual similarity analysis, use the existing advanced script:
+```bash
+cd apps/multimodal/vision-based-pdf-multi-vector/
+python multi-vector-leann-similarity-map.py
+```
+
+Edit the script to customize:
+- `QUERY`: Your question
+- `MODEL`: "colqwen2" or "colpali"
+- `USE_HF_DATASET`: Use HuggingFace dataset or local PDFs
+- `SIMILARITY_MAP`: Generate heatmaps
+- `ANSWER`: Enable Qwen-VL answer generation
+
+## 🔧 How It Works
+
+### ColQwen2 vs ColPali
+- **ColQwen2** (`vidore/colqwen2-v1.0`): Latest vision-language model
+- **ColPali** (`vidore/colpali-v1.2`): Proven multimodal retriever
+
+### Architecture
+1. **PDF → Images**: Convert PDF pages to images (150 DPI)
+2. **Vision Encoding**: Process images with ColQwen2/ColPali
+3. **Multi-Vector Index**: Build LEANN HNSW index with multiple embeddings per page
+4. **Query Processing**: Encode text queries with same model
+5. **Similarity Search**: Find most relevant pages/regions
+6. **Visual Maps**: Generate attention heatmaps (optional)
+
+### Device Support
+- **CUDA**: Best performance with GPU acceleration
+- **MPS**: Apple Silicon Mac support
+- **CPU**: Fallback for any system (slower)
+
+Auto-detection: CUDA > MPS > CPU
+
+## 📊 Performance Tips
+
+### For Best Performance:
+```bash
+# Use ColQwen2 for latest features
+--model colqwen2
+
+# Save page images for reuse
+--pages-dir ./cached_pages/
+
+# Adjust batch size based on GPU memory
+# (automatically handled)
+```
+
+### For Large Document Sets:
+- Process PDFs in batches
+- Use SSD storage for index files
+- Consider using CUDA if available
+
+## 🔗 Related Resources
+
+- **Fast-PLAID**: https://github.com/lightonai/fast-plaid
+- **Pylate**: https://github.com/lightonai/pylate
+- **ColBERT**: https://github.com/stanford-futuredata/ColBERT
+- **ColPali Paper**: Vision-Language Models for Document Retrieval
+- **Issue #119**: https://github.com/yichuan-w/LEANN/issues/119
+
+## 🐛 Troubleshooting
+
+### PDF Conversion Issues (macOS)
+```bash
+# Install poppler
+brew install poppler
+which pdfinfo && pdfinfo -v
+```
+
+### Memory Issues
+- Reduce batch size (automatically handled)
+- Use CPU instead of GPU: `export CUDA_VISIBLE_DEVICES=""`
+- Process fewer PDFs at once
+
+### Model Download Issues
+- Ensure internet connection for first run
+- Models are cached after first download
+- Use HuggingFace mirrors if needed
+
+### Import Errors
+```bash
+# Ensure all dependencies installed
+uv pip install colpali_engine pdf2image pillow matplotlib qwen_vl_utils einops seaborn
+
+# Check PyTorch installation
+python -c "import torch; print(torch.__version__)"
+```
+
+## 💡 Examples
+
+### Research Paper Analysis
+```bash
+# Index your research papers
+python -m apps.colqwen_rag build --pdfs ~/Papers/AI/ --index ai_papers
+
+# Ask research questions
+python -m apps.colqwen_rag search ai_papers "What are the limitations of transformer models?"
+python -m apps.colqwen_rag search ai_papers "How does BERT compare to GPT?"
+```
+
+### Document Q&A
+```bash
+# Index business documents
+python -m apps.colqwen_rag build --pdfs ~/Documents/Reports/ --index reports
+
+# Interactive analysis
+python -m apps.colqwen_rag ask reports --interactive
+```
+
+### Visual Analysis
+```bash
+# Generate similarity maps for specific queries
+cd apps/multimodal/vision-based-pdf-multi-vector/
+# Edit multi-vector-leann-similarity-map.py with your query
+python multi-vector-leann-similarity-map.py
+# Check ./figures/ for generated heatmaps
+```
+
+---
+
+**🎯 This integration makes ColQwen as easy to use as other LEANN features while maintaining the full power of multimodal document understanding!**

docs/CONTRIBUTING.md

@@ -0,0 +1,224 @@
+# 🤝 Contributing
+
+We welcome contributions! Leann is built by the community, for the community.
+
+## Ways to Contribute
+
+- 🐛 **Bug Reports**: Found an issue? Let us know!
+- 💡 **Feature Requests**: Have an idea? We'd love to hear it!
+- 🔧 **Code Contributions**: PRs welcome for all skill levels
+- 📖 **Documentation**: Help make Leann more accessible
+- 🧪 **Benchmarks**: Share your performance results
+
+## 🚀 Development Setup
+
+### Prerequisites
+
+1. **Install uv** (fast Python package installer):
+   ```bash
+   curl -LsSf https://astral.sh/uv/install.sh | sh
+   ```
+
+2. **Clone the repository**:
+   ```bash
+   git clone https://github.com/yichuan-w/LEANN.git leann
+   git submodule update --init --recursive
+   cd leann
+   ```
+
+3. **Install system dependencies**:
+
+   **macOS:**
+   ```bash
+   brew install llvm libomp boost protobuf zeromq pkgconf
+   ```
+
+   **Ubuntu/Debian:**
+   ```bash
+   sudo apt-get install libomp-dev libboost-all-dev protobuf-compiler \
+                        libabsl-dev libmkl-full-dev libaio-dev libzmq3-dev
+   ```
+
+4. **Build from source**:
+   ```bash
+   # macOS
+   CC=$(brew --prefix llvm)/bin/clang CXX=$(brew --prefix llvm)/bin/clang++ uv sync
+
+   # Ubuntu/Debian
+   uv sync
+   ```
+
+## 🔨 Pre-commit Hooks
+
+We use pre-commit hooks to ensure code quality and consistency. This runs automatically before each commit.
+
+### Setup Pre-commit
+
+1. **Install pre-commit tools**:
+   ```bash
+   uv sync --group lint
+   ```
+
+2. **Install the git hooks**:
+   ```bash
+   pre-commit install
+   ```
+
+3. **Run pre-commit manually** (optional):
+   ```bash
+   uv run pre-commit run --all-files
+   ```
+
+### Pre-commit Checks
+
+Our pre-commit configuration includes:
+- **Trailing whitespace removal**
+- **End-of-file fixing**
+- **YAML validation**
+- **Large file prevention**
+- **Merge conflict detection**
+- **Debug statement detection**
+- **Code formatting with ruff**
+- **Code linting with ruff**
+
+## 🧪 Testing
+
+### Running Tests
+
+```bash
+# Install test tools only (no project runtime)
+uv sync --group test
+
+# Run all tests
+uv run pytest
+
+# Run specific test file
+uv run pytest test/test_filename.py
+
+# Run with coverage
+uv run pytest --cov=leann
+```
+
+### Writing Tests
+
+- Place tests in the `test/` directory
+- Follow the naming convention `test_*.py`
+- Use descriptive test names that explain what's being tested
+- Include both positive and negative test cases
+
+## 📝 Code Style
+
+We use `ruff` for both linting and formatting to ensure consistent code style.
+
+### Format Your Code
+
+```bash
+# Format all files
+ruff format
+
+# Check formatting without changing files
+ruff format --check
+```
+
+### Lint Your Code
+
+```bash
+# Run linter with auto-fix
+ruff check --fix
+
+# Just check without fixing
+ruff check
+```
+
+### Style Guidelines
+
+- Follow PEP 8 conventions
+- Use descriptive variable names
+- Add type hints where appropriate
+- Write docstrings for all public functions and classes
+- Keep functions focused and single-purpose
+
+## 🚦 CI/CD
+
+Our CI pipeline runs automatically on all pull requests. It includes:
+
+1. **Linting and Formatting**: Ensures code follows our style guidelines
+2. **Multi-platform builds**: Tests on Ubuntu and macOS
+3. **Python version matrix**: Tests on Python 3.9-3.13
+4. **Wheel building**: Ensures packages can be built and distributed
+
+### CI Commands
+
+The CI uses the same commands as pre-commit to ensure consistency:
+```bash
+# Linting
+ruff check .
+
+# Format checking
+ruff format --check .
+```
+
+Make sure your code passes these checks locally before pushing!
+
+## 🔄 Pull Request Process
+
+1. **Fork the repository** and create your branch from `main`:
+   ```bash
+   git checkout -b feature/your-feature-name
+   ```
+
+2. **Make your changes**:
+   - Write clean, documented code
+   - Add tests for new functionality
+   - Update documentation as needed
+
+3. **Run pre-commit checks**:
+   ```bash
+   pre-commit run --all-files
+   ```
+
+4. **Test your changes**:
+   ```bash
+   uv run pytest
+   ```
+
+5. **Commit with descriptive messages**:
+   ```bash
+   git commit -m "feat: add new search algorithm"
+   ```
+
+   Follow [Conventional Commits](https://www.conventionalcommits.org/):
+   - `feat:` for new features
+   - `fix:` for bug fixes
+   - `docs:` for documentation changes
+   - `test:` for test additions/changes
+   - `refactor:` for code refactoring
+   - `perf:` for performance improvements
+
+6. **Push and create a pull request**:
+   - Provide a clear description of your changes
+   - Reference any related issues
+   - Include examples or screenshots if applicable
+
+## 📚 Documentation
+
+When adding new features or making significant changes:
+
+1. Update relevant documentation in `/docs`
+2. Add docstrings to new functions/classes
+3. Update README.md if needed
+4. Include usage examples
+
+## 🤔 Getting Help
+
+- **Discord**: Join our community for discussions
+- **Issues**: Check existing issues or create a new one
+- **Discussions**: For general questions and ideas
+
+## 📄 License
+
+By contributing, you agree that your contributions will be licensed under the same license as the project (MIT).
+
+---
+
+Thank you for contributing to LEANN! Every contribution, no matter how small, helps make the project better for everyone. 🌟

docs/RELEASE.md

@@ -0,0 +1,22 @@
+# Release Guide
+
+## Setup (One-time)
+
+Add `PYPI_API_TOKEN` to GitHub Secrets:
+1. Get token: https://pypi.org/manage/account/token/
+2. Add to secrets: Settings → Secrets → Actions → `PYPI_API_TOKEN`
+
+## Release (One-click)
+
+1. Go to: https://github.com/yichuan-w/LEANN/actions/workflows/release-manual.yml
+2. Click "Run workflow"
+3. Enter version: `0.1.2`
+4. Click green "Run workflow" button
+
+That's it! The workflow will automatically:
+- ✅ Update version in all packages
+- ✅ Build all packages
+- ✅ Publish to PyPI
+- ✅ Create GitHub tag and release
+
+Check progress: https://github.com/yichuan-w/LEANN/actions

docs/THINKING_BUDGET_FEATURE.md

@@ -0,0 +1,123 @@
+# Thinking Budget Feature Implementation
+
+## Overview
+
+This document describes the implementation of the **thinking budget** feature for LEANN, which allows users to control the computational effort for reasoning models like GPT-Oss:20b.
+
+## Feature Description
+
+The thinking budget feature provides three levels of computational effort for reasoning models:
+- **`low`**: Fast responses, basic reasoning (default for simple queries)
+- **`medium`**: Balanced speed and reasoning depth
+- **`high`**: Maximum reasoning effort, best for complex analytical questions
+
+## Implementation Details
+
+### 1. Command Line Interface
+
+Added `--thinking-budget` parameter to both CLI and RAG examples:
+
+```bash
+# LEANN CLI
+leann ask my-index --llm ollama --model gpt-oss:20b --thinking-budget high
+
+# RAG Examples
+python apps/email_rag.py --llm ollama --llm-model gpt-oss:20b --thinking-budget high
+python apps/document_rag.py --llm openai --llm-model o3 --thinking-budget medium
+```
+
+### 2. LLM Backend Support
+
+#### Ollama Backend (`packages/leann-core/src/leann/chat.py`)
+
+```python
+def ask(self, prompt: str, **kwargs) -> str:
+    # Handle thinking budget for reasoning models
+    options = kwargs.copy()
+    thinking_budget = kwargs.get("thinking_budget")
+    if thinking_budget:
+        options.pop("thinking_budget", None)
+        if thinking_budget in ["low", "medium", "high"]:
+            options["reasoning"] = {"effort": thinking_budget, "exclude": False}
+```
+
+**API Format**: Uses Ollama's `reasoning` parameter with `effort` and `exclude` fields.
+
+#### OpenAI Backend (`packages/leann-core/src/leann/chat.py`)
+
+```python
+def ask(self, prompt: str, **kwargs) -> str:
+    # Handle thinking budget for reasoning models
+    thinking_budget = kwargs.get("thinking_budget")
+    if thinking_budget and thinking_budget in ["low", "medium", "high"]:
+        # Check if this is an o-series model
+        o_series_models = ["o3", "o3-mini", "o4-mini", "o1", "o3-pro", "o3-deep-research"]
+        if any(model in self.model for model in o_series_models):
+            params["reasoning_effort"] = thinking_budget
+```
+
+**API Format**: Uses OpenAI's `reasoning_effort` parameter for o-series models.
+
+### 3. Parameter Propagation
+
+The thinking budget parameter is properly propagated through the LEANN architecture:
+
+1. **CLI** (`packages/leann-core/src/leann/cli.py`): Captures `--thinking-budget` argument
+2. **Base RAG** (`apps/base_rag_example.py`): Adds parameter to argument parser
+3. **LeannChat** (`packages/leann-core/src/leann/api.py`): Passes `llm_kwargs` to LLM
+4. **LLM Interface**: Handles the parameter in backend-specific implementations
+
+## Files Modified
+
+### Core Implementation
+- `packages/leann-core/src/leann/chat.py`: Added thinking budget support to OllamaChat and OpenAIChat
+- `packages/leann-core/src/leann/cli.py`: Added `--thinking-budget` argument
+- `apps/base_rag_example.py`: Added thinking budget parameter to RAG examples
+
+### Documentation
+- `README.md`: Added thinking budget parameter to usage examples
+- `docs/configuration-guide.md`: Added detailed documentation and usage guidelines
+
+### Examples
+- `examples/thinking_budget_demo.py`: Comprehensive demo script with usage examples
+
+## Usage Examples
+
+### Basic Usage
+```bash
+# High reasoning effort for complex questions
+leann ask my-index --llm ollama --model gpt-oss:20b --thinking-budget high
+
+# Medium reasoning for balanced performance
+leann ask my-index --llm openai --model gpt-4o --thinking-budget medium
+
+# Low reasoning for fast responses
+leann ask my-index --llm ollama --model gpt-oss:20b --thinking-budget low
+```
+
+### RAG Examples
+```bash
+# Email RAG with high reasoning
+python apps/email_rag.py --llm ollama --llm-model gpt-oss:20b --thinking-budget high
+
+# Document RAG with medium reasoning
+python apps/document_rag.py --llm openai --llm-model gpt-4o --thinking-budget medium
+```
+
+## Supported Models
+
+### Ollama Models
+- **GPT-Oss:20b**: Primary target model with reasoning capabilities
+- **Other reasoning models**: Any Ollama model that supports the `reasoning` parameter
+
+### OpenAI Models
+- **o3, o3-mini, o4-mini, o1**: o-series reasoning models with `reasoning_effort` parameter
+- **GPT-OSS models**: Models that support reasoning capabilities
+
+## Testing
+
+The implementation includes comprehensive testing:
+- Parameter handling verification
+- Backend-specific API format validation
+- CLI argument parsing tests
+- Integration with existing LEANN architecture

docs/ast_chunking_guide.md

@@ -0,0 +1,143 @@
+# AST-Aware Code chunking guide
+
+## Overview
+
+This guide covers best practices for using AST-aware code chunking in LEANN. AST chunking provides better semantic understanding of code structure compared to traditional text-based chunking.
+
+## Quick Start
+
+### Basic Usage
+
+```bash
+# Enable AST chunking for mixed content (code + docs)
+python -m apps.document_rag --enable-code-chunking --data-dir ./my_project
+
+# Specialized code repository indexing
+python -m apps.code_rag --repo-dir ./my_codebase
+
+# Global CLI with AST support
+leann build my-code-index --docs ./src --use-ast-chunking
+```
+
+### Installation
+
+```bash
+# Install LEANN with AST chunking support
+uv pip install -e "."
+```
+
+#### For normal users (PyPI install)
+- Use `pip install leann` or `uv pip install leann`.
+- `astchunk` is pulled automatically from PyPI as a dependency; no extra steps.
+
+#### For developers (from source, editable)
+```bash
+git clone https://github.com/yichuan-w/LEANN.git leann
+cd leann
+git submodule update --init --recursive
+uv sync
+```
+- This repo vendors `astchunk` as a git submodule at `packages/astchunk-leann` (our fork).
+- `[tool.uv.sources]` maps the `astchunk` package to that path in editable mode.
+- You can edit code under `packages/astchunk-leann` and Python will use your changes immediately (no separate `pip install astchunk` needed).
+
+## Best Practices
+
+### When to Use AST Chunking
+
+✅ **Recommended for:**
+- Code repositories with multiple languages
+- Mixed documentation and code content
+- Complex codebases with deep function/class hierarchies
+- When working with Claude Code for code assistance
+
+❌ **Not recommended for:**
+- Pure text documents
+- Very large files (>1MB)
+- Languages not supported by tree-sitter
+
+### Optimal Configuration
+
+```bash
+# Recommended settings for most codebases
+python -m apps.code_rag \
+    --repo-dir ./src \
+    --ast-chunk-size 768 \
+    --ast-chunk-overlap 96 \
+    --exclude-dirs .git __pycache__ node_modules build dist
+```
+
+### Supported Languages
+
+| Extension | Language | Status |
+|-----------|----------|--------|
+| `.py` | Python | ✅ Full support |
+| `.java` | Java | ✅ Full support |
+| `.cs` | C# | ✅ Full support |
+| `.ts`, `.tsx` | TypeScript | ✅ Full support |
+| `.js`, `.jsx` | JavaScript | ✅ Via TypeScript parser |
+
+## Integration Examples
+
+### Document RAG with Code Support
+
+```python
+# Enable code chunking in document RAG
+python -m apps.document_rag \
+    --enable-code-chunking \
+    --data-dir ./project \
+    --query "How does authentication work in the codebase?"
+```
+
+### Claude Code Integration
+
+When using with Claude Code MCP server, AST chunking provides better context for:
+- Code completion and suggestions
+- Bug analysis and debugging
+- Architecture understanding
+- Refactoring assistance
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Fallback to Traditional Chunking**
+   - Normal behavior for unsupported languages
+   - Check logs for specific language support
+
+2. **Performance with Large Files**
+   - Adjust `--max-file-size` parameter
+   - Use `--exclude-dirs` to skip unnecessary directories
+
+3. **Quality Issues**
+   - Try different `--ast-chunk-size` values (512, 768, 1024)
+   - Adjust overlap for better context preservation
+
+### Debug Mode
+
+```bash
+export LEANN_LOG_LEVEL=DEBUG
+python -m apps.code_rag --repo-dir ./my_code
+```
+
+## Migration from Traditional Chunking
+
+Existing workflows continue to work without changes. To enable AST chunking:
+
+```bash
+# Before
+python -m apps.document_rag --chunk-size 256
+
+# After (maintains traditional chunking for non-code files)
+python -m apps.document_rag --enable-code-chunking --chunk-size 256 --ast-chunk-size 768
+```
+
+## References
+
+- [astchunk GitHub Repository](https://github.com/yilinjz/astchunk)
+- [LEANN MCP Integration](../packages/leann-mcp/README.md)
+- [Research Paper](https://arxiv.org/html/2506.15655v1)
+
+---
+
+**Note**: AST chunking maintains full backward compatibility while enhancing code understanding capabilities.

docs/code/embedding_model_compare.py

@@ -0,0 +1,98 @@
+"""
+Comparison between Sentence Transformers and OpenAI embeddings
+
+This example shows how different embedding models handle complex queries
+and demonstrates the differences between local and API-based embeddings.
+"""
+
+import numpy as np
+from leann.embedding_compute import compute_embeddings
+
+# OpenAI API key should be set as environment variable
+# export OPENAI_API_KEY="your-api-key-here"
+
+# Test data
+conference_text = "[Title]: COLING 2025 Conference\n[URL]: https://coling2025.org/"
+browser_text = "[Title]: Browser Use Tool\n[URL]: https://github.com/browser-use"
+
+# Two queries with same intent but different wording
+query1 = "Tell me my browser history about some conference i often visit"
+query2 = "browser history about conference I often visit"
+
+texts = [query1, query2, conference_text, browser_text]
+
+
+def cosine_similarity(a, b):
+    return np.dot(a, b)  # Already normalized
+
+
+def analyze_embeddings(embeddings, model_name):
+    print(f"\n=== {model_name} Results ===")
+
+    # Results for Query 1
+    sim1_conf = cosine_similarity(embeddings[0], embeddings[2])
+    sim1_browser = cosine_similarity(embeddings[0], embeddings[3])
+
+    print(f"Query 1: '{query1}'")
+    print(f"  → Conference similarity: {sim1_conf:.4f} {'✓' if sim1_conf > sim1_browser else ''}")
+    print(
+        f"  → Browser similarity:    {sim1_browser:.4f} {'✓' if sim1_browser > sim1_conf else ''}"
+    )
+    print(f"  Winner: {'Conference' if sim1_conf > sim1_browser else 'Browser'}")
+
+    # Results for Query 2
+    sim2_conf = cosine_similarity(embeddings[1], embeddings[2])
+    sim2_browser = cosine_similarity(embeddings[1], embeddings[3])
+
+    print(f"\nQuery 2: '{query2}'")
+    print(f"  → Conference similarity: {sim2_conf:.4f} {'✓' if sim2_conf > sim2_browser else ''}")
+    print(
+        f"  → Browser similarity:    {sim2_browser:.4f} {'✓' if sim2_browser > sim2_conf else ''}"
+    )
+    print(f"  Winner: {'Conference' if sim2_conf > sim2_browser else 'Browser'}")
+
+    # Show the impact
+    print("\n=== Impact Analysis ===")
+    print(f"Conference similarity change: {sim2_conf - sim1_conf:+.4f}")
+    print(f"Browser similarity change:    {sim2_browser - sim1_browser:+.4f}")
+
+    if sim1_conf > sim1_browser and sim2_browser > sim2_conf:
+        print("❌ FLIP: Adding 'browser history' flips winner from Conference to Browser!")
+    elif sim1_conf > sim1_browser and sim2_conf > sim2_browser:
+        print("✅ STABLE: Conference remains winner in both queries")
+    elif sim1_browser > sim1_conf and sim2_browser > sim2_conf:
+        print("✅ STABLE: Browser remains winner in both queries")
+    else:
+        print("🔄 MIXED: Results vary between queries")
+
+    return {
+        "query1_conf": sim1_conf,
+        "query1_browser": sim1_browser,
+        "query2_conf": sim2_conf,
+        "query2_browser": sim2_browser,
+    }
+
+
+# Test Sentence Transformers
+print("Testing Sentence Transformers (facebook/contriever)...")
+try:
+    st_embeddings = compute_embeddings(texts, "facebook/contriever", mode="sentence-transformers")
+    st_results = analyze_embeddings(st_embeddings, "Sentence Transformers (facebook/contriever)")
+except Exception as e:
+    print(f"❌ Sentence Transformers failed: {e}")
+    st_results = None
+
+# Test OpenAI
+print("\n" + "=" * 60)
+print("Testing OpenAI (text-embedding-3-small)...")
+try:
+    openai_embeddings = compute_embeddings(texts, "text-embedding-3-small", mode="openai")
+    openai_results = analyze_embeddings(openai_embeddings, "OpenAI (text-embedding-3-small)")
+except Exception as e:
+    print(f"❌ OpenAI failed: {e}")
+    openai_results = None
+
+# Compare results
+if st_results and openai_results:
+    print("\n" + "=" * 60)
+    print("=== COMPARISON SUMMARY ===")

docs/configuration-guide.md

@@ -0,0 +1,563 @@
+# LEANN Configuration Guide
+
+This guide helps you optimize LEANN for different use cases and understand the trade-offs between various configuration options.
+
+## Getting Started: Simple is Better
+
+When first trying LEANN, start with a small dataset to quickly validate your approach:
+
+**For document RAG**: The default `data/` directory works perfectly - includes 2 AI research papers, Pride and Prejudice literature, and a technical report
+```bash
+python -m apps.document_rag --query "What techniques does LEANN use?"
+```
+
+**For other data sources**: Limit the dataset size for quick testing
+```bash
+# WeChat: Test with recent messages only
+python -m apps.wechat_rag --max-items 100 --query "What did we discuss about the project timeline?"
+
+# Browser history: Last few days
+python -m apps.browser_rag --max-items 500 --query "Find documentation about vector databases"
+
+# Email: Recent inbox
+python -m apps.email_rag --max-items 200 --query "Who sent updates about the deployment status?"
+```
+
+Once validated, scale up gradually:
+- 100 documents → 1,000 → 10,000 → full dataset (`--max-items -1`)
+- This helps identify issues early before committing to long processing times
+
+## Embedding Model Selection: Understanding the Trade-offs
+
+Based on our experience developing LEANN, embedding models fall into three categories:
+
+### Small Models (< 100M parameters)
+**Example**: `sentence-transformers/all-MiniLM-L6-v2` (22M params)
+- **Pros**: Lightweight, fast for both indexing and inference
+- **Cons**: Lower semantic understanding, may miss nuanced relationships
+- **Use when**: Speed is critical, handling simple queries, interactive mode, or just experimenting with LEANN. If time is not a constraint, consider using a larger/better embedding model
+
+### Medium Models (100M-500M parameters)
+**Example**: `facebook/contriever` (110M params), `BAAI/bge-base-en-v1.5` (110M params)
+- **Pros**: Balanced performance, good multilingual support, reasonable speed
+- **Cons**: Requires more compute than small models
+- **Use when**: Need quality results without extreme compute requirements, general-purpose RAG applications
+
+### Large Models (500M+ parameters)
+**Example**: `Qwen/Qwen3-Embedding-0.6B` (600M params), `intfloat/multilingual-e5-large` (560M params)
+- **Pros**: Best semantic understanding, captures complex relationships, excellent multilingual support. **Qwen3-Embedding-0.6B achieves nearly OpenAI API performance!**
+- **Cons**: Slower inference, longer index build times
+- **Use when**: Quality is paramount and you have sufficient compute resources. **Highly recommended** for production use
+
+### Quick Start: Cloud and Local Embedding Options
+
+**OpenAI Embeddings (Fastest Setup)**
+For immediate testing without local model downloads(also if you [do not have GPU](https://github.com/yichuan-w/LEANN/issues/43) and do not care that much about your document leak, you should use this, we compute the embedding and recompute using openai API):
+```bash
+# Set OpenAI embeddings (requires OPENAI_API_KEY)
+--embedding-mode openai --embedding-model text-embedding-3-small
+```
+
+**Ollama Embeddings (Privacy-Focused)**
+For local embeddings with complete privacy:
+```bash
+# First, pull an embedding model
+ollama pull nomic-embed-text
+
+# Use Ollama embeddings
+--embedding-mode ollama --embedding-model nomic-embed-text
+```
+
+<details>
+<summary><strong>Cloud vs Local Trade-offs</strong></summary>
+
+**OpenAI Embeddings** (`text-embedding-3-small/large`)
+- **Pros**: No local compute needed, consistently fast, high quality
+- **Cons**: Requires API key, costs money, data leaves your system, [known limitations with certain languages](https://yichuan-w.github.io/blog/lessons_learned_in_dev_leann/)
+- **When to use**: Prototyping, non-sensitive data, need immediate results
+
+**Local Embeddings**
+- **Pros**: Complete privacy, no ongoing costs, full control, can sometimes outperform OpenAI embeddings
+- **Cons**: Slower than cloud APIs, requires local compute resources
+- **When to use**: Production systems, sensitive data, cost-sensitive applications
+
+</details>
+
+## Local & Remote Inference Endpoints
+
+> Applies to both LLMs (`leann ask`) and embeddings (`leann build`).
+
+LEANN now treats Ollama, LM Studio, and other OpenAI-compatible runtimes as first-class providers. You can point LEANN at any compatible endpoint – either on the same machine or across the network – with a couple of flags or environment variables.
+
+### One-Time Environment Setup
+
+```bash
+# Works for OpenAI-compatible runtimes such as LM Studio, vLLM, SGLang, llamafile, etc.
+export OPENAI_API_KEY="your-key"            # or leave unset for local servers that do not check keys
+export OPENAI_BASE_URL="http://localhost:1234/v1"
+
+# Ollama-compatible runtimes (Ollama, Ollama on another host, llamacpp-server, etc.)
+export LEANN_OLLAMA_HOST="http://localhost:11434"   # falls back to OLLAMA_HOST or LOCAL_LLM_ENDPOINT
+```
+
+LEANN also recognises `LEANN_LOCAL_LLM_HOST` (highest priority), `LEANN_OPENAI_BASE_URL`, and `LOCAL_OPENAI_BASE_URL`, so existing scripts continue to work.
+
+### Passing Hosts Per Command
+
+```bash
+# Build an index with a remote embedding server
+leann build my-notes \
+  --docs ./notes \
+  --embedding-mode openai \
+  --embedding-model text-embedding-qwen3-embedding-0.6b \
+  --embedding-api-base http://192.168.1.50:1234/v1 \
+  --embedding-api-key local-dev-key
+
+# Query using a local LM Studio instance via OpenAI-compatible API
+leann ask my-notes \
+  --llm openai \
+  --llm-model qwen3-8b \
+  --api-base http://localhost:1234/v1 \
+  --api-key local-dev-key
+
+# Query an Ollama instance running on another box
+leann ask my-notes \
+  --llm ollama \
+  --llm-model qwen3:14b \
+  --host http://192.168.1.101:11434
+```
+
+⚠️ **Make sure the endpoint is reachable**: when your inference server runs on a home/workstation and the index/search job runs in the cloud, the server must be able to reach the host you configured. Typical options include:
+
+- Expose a public IP (and open the relevant port) on the machine that hosts LM Studio/Ollama.
+- Configure router or cloud provider port forwarding.
+- Tunnel traffic through tools like `tailscale`, `cloudflared`, or `ssh -R`.
+
+When you set these options while building an index, LEANN stores them in `meta.json`. Any subsequent `leann ask` or searcher process automatically reuses the same provider settings – even when we spawn background embedding servers. This makes the “server without GPU talking to my local workstation” workflow from [issue #80](https://github.com/yichuan-w/LEANN/issues/80#issuecomment-2287230548) work out-of-the-box.
+
+**Tip:** If your runtime does not require an API key (many local stacks don’t), leave `--api-key` unset. LEANN will skip injecting credentials.
+
+### Python API Usage
+
+You can pass the same configuration from Python:
+
+```python
+from leann.api import LeannBuilder
+
+builder = LeannBuilder(
+    backend_name="hnsw",
+    embedding_mode="openai",
+    embedding_model="text-embedding-qwen3-embedding-0.6b",
+    embedding_options={
+        "base_url": "http://192.168.1.50:1234/v1",
+        "api_key": "local-dev-key",
+    },
+)
+builder.build_index("./indexes/my-notes", chunks)
+```
+
+`embedding_options` is persisted to the index `meta.json`, so subsequent `LeannSearcher` or `LeannChat` sessions automatically reuse the same provider settings (the embedding server manager forwards them to the provider for you).
+
+## Optional Embedding Features
+
+### Task-Specific Prompt Templates
+
+Some embedding models are trained with task-specific prompts to differentiate between documents and queries. The most notable example is **Google's EmbeddingGemma**, which requires different prompts depending on the use case:
+
+- **Indexing documents**: `"title: none | text: "`
+- **Search queries**: `"task: search result | query: "`
+
+LEANN supports automatic prompt prepending via the `--embedding-prompt-template` flag:
+
+```bash
+# Build index with EmbeddingGemma (via LM Studio or Ollama)
+leann build my-docs \
+  --docs ./documents \
+  --embedding-mode openai \
+  --embedding-model text-embedding-embeddinggemma-300m-qat \
+  --embedding-api-base http://localhost:1234/v1 \
+  --embedding-prompt-template "title: none | text: " \
+  --force
+
+# Search with query-specific prompt
+leann search my-docs \
+  --query "What is quantum computing?" \
+  --embedding-prompt-template "task: search result | query: "
+```
+
+A full example that is used for building the LEANN's repo during dev:
+```
+source "$LEANN_PATH/.venv/bin/activate" && \
+leann build --docs $(git ls-files | grep -Ev '\.(png|jpg|jpeg|gif|yml|yaml|sh|pdf|JPG)$') --embedding-mode openai \
+--embedding-model text-embedding-embeddinggemma-300m-qat \
+--embedding-prompt-template "title: none | text: " \
+--query-prompt-template "task: search result | query: " \
+--embedding-api-key local-dev-key \
+--embedding-api-base http://localhost:1234/v1 \
+--doc-chunk-size 1024 --doc-chunk-overlap 100 \
+--code-chunk-size 1024 --code-chunk-overlap 100 \
+--ast-chunk-size 1024 --ast-chunk-overlap 100 \
+--force --use-ast-chunking --no-compact --no-recompute
+```
+
+**Important Notes:**
+- **Only use with compatible models**: EmbeddingGemma and similar task-specific models
+- **NOT for regular models**: Adding prompts to models like `nomic-embed-text`, `text-embedding-3-small`, or `bge-base-en-v1.5` will corrupt embeddings
+- **Template is saved**: Build-time templates are saved to `.meta.json` for reference; you can add both `--embedding-prompt-template` and `--query-prompt-template` values during building phase, and this way the mcp query will automatically pick up the query template
+- **Flexible prompts**: You can use any prompt string, or leave it empty (`""`)
+
+**Python API:**
+```python
+from leann.api import LeannBuilder
+
+builder = LeannBuilder(
+    embedding_mode="openai",
+    embedding_model="text-embedding-embeddinggemma-300m-qat",
+    embedding_options={
+        "base_url": "http://localhost:1234/v1",
+        "api_key": "lm-studio",
+        "prompt_template": "title: none | text: ",
+    },
+)
+builder.build_index("./indexes/my-docs", chunks)
+```
+
+**References:**
+- [HuggingFace Blog: EmbeddingGemma](https://huggingface.co/blog/embeddinggemma) - Technical details
+
+### LM Studio Auto-Detection (Optional)
+
+When using LM Studio with the OpenAI-compatible API, LEANN can optionally auto-detect model context lengths via the LM Studio SDK. This eliminates manual configuration for token limits.
+
+**Prerequisites:**
+```bash
+# Install Node.js (if not already installed)
+# Then install the LM Studio SDK globally
+npm install -g @lmstudio/sdk
+```
+
+**How it works:**
+1. LEANN detects LM Studio URLs (`:1234`, `lmstudio` in URL)
+2. Queries model metadata via Node.js subprocess
+3. Automatically unloads model after query (respects your JIT auto-evict settings)
+4. Falls back to static registry if SDK unavailable
+
+**No configuration needed** - it works automatically when SDK is installed:
+
+```bash
+leann build my-docs \
+  --docs ./documents \
+  --embedding-mode openai \
+  --embedding-model text-embedding-nomic-embed-text-v1.5 \
+  --embedding-api-base http://localhost:1234/v1
+  # Context length auto-detected if SDK available
+  # Falls back to registry (2048) if not
+```
+
+**Benefits:**
+- ✅ Automatic token limit detection
+- ✅ Respects LM Studio JIT auto-evict settings
+- ✅ No manual registry maintenance
+- ✅ Graceful fallback if SDK unavailable
+
+**Note:** This is completely optional. LEANN works perfectly fine without the SDK using the built-in token limit registry.
+
+## Index Selection: Matching Your Scale
+
+### HNSW (Hierarchical Navigable Small World)
+**Best for**: Small to medium datasets (< 10M vectors) - **Default and recommended for extreme low storage**
+- Full recomputation required
+- High memory usage during build phase
+- Excellent recall (95%+)
+
+```bash
+# Optimal for most use cases
+--backend-name hnsw --graph-degree 32 --build-complexity 64
+```
+
+### DiskANN
+**Best for**: Large datasets, especially when you want `recompute=True`.
+
+**Key advantages:**
+- **Faster search** on large datasets (3x+ speedup vs HNSW in many cases)
+- **Smart storage**: `recompute=True` enables automatic graph partitioning for smaller indexes
+- **Better scaling**: Designed for 100k+ documents
+
+**Recompute behavior:**
+- `recompute=True` (recommended): Pure PQ traversal + final reranking - faster and enables partitioning
+- `recompute=False`: PQ + partial real distances during traversal - slower but higher accuracy
+
+```bash
+# Recommended for most use cases
+--backend-name diskann --graph-degree 32 --build-complexity 64
+```
+
+**Performance Benchmark**: Run `uv run benchmarks/diskann_vs_hnsw_speed_comparison.py` to compare DiskANN and HNSW on your system.
+
+## LLM Selection: Engine and Model Comparison
+
+### LLM Engines
+
+**OpenAI** (`--llm openai`)
+- **Pros**: Best quality, consistent performance, no local resources needed
+- **Cons**: Costs money ($0.15-2.5 per million tokens), requires internet, data privacy concerns
+- **Models**: `gpt-4o-mini` (fast, cheap), `gpt-4o` (best quality), `o3` (reasoning), `o3-mini` (reasoning, cheaper)
+- **Thinking Budget**: Use `--thinking-budget low/medium/high` for o-series reasoning models (o3, o3-mini, o4-mini)
+- **Note**: Our current default, but we recommend switching to Ollama for most use cases
+
+**Ollama** (`--llm ollama`)
+- **Pros**: Fully local, free, privacy-preserving, good model variety
+- **Cons**: Requires local GPU/CPU resources, slower than cloud APIs, need to install extra [ollama app](https://github.com/ollama/ollama?tab=readme-ov-file#ollama) and pre-download models by `ollama pull`
+- **Models**: `qwen3:0.6b` (ultra-fast), `qwen3:1.7b` (balanced), `qwen3:4b` (good quality), `qwen3:7b` (high quality), `deepseek-r1:1.5b` (reasoning)
+- **Thinking Budget**: Use `--thinking-budget low/medium/high` for reasoning models like GPT-Oss:20b
+
+**HuggingFace** (`--llm hf`)
+- **Pros**: Free tier available, huge model selection, direct model loading (vs Ollama's server-based approach)
+- **Cons**: More complex initial setup
+- **Models**: `Qwen/Qwen3-1.7B-FP8`
+
+## Parameter Tuning Guide
+
+### Search Complexity Parameters
+
+**`--build-complexity`** (index building)
+- Controls thoroughness during index construction
+- Higher = better recall but slower build
+- Recommendations:
+  - 32: Quick prototyping
+  - 64: Balanced (default)
+  - 128: Production systems
+  - 256: Maximum quality
+
+**`--search-complexity`** (query time)
+- Controls search thoroughness
+- Higher = better results but slower
+- Recommendations:
+  - 16: Fast/Interactive search
+  - 32: High quality with diversity
+  - 64+: Maximum accuracy
+
+### Top-K Selection
+
+**`--top-k`** (number of retrieved chunks)
+- More chunks = better context but slower LLM processing
+- Should be always smaller than `--search-complexity`
+- Guidelines:
+  - 10-20: General questions (default: 20)
+  - 30+: Complex multi-hop reasoning requiring comprehensive context
+
+**Trade-off formula**:
+- Retrieval time ∝ log(n) × search_complexity
+- LLM processing time ∝ top_k × chunk_size
+- Total context = top_k × chunk_size tokens
+
+### Thinking Budget for Reasoning Models
+
+**`--thinking-budget`** (reasoning effort level)
+- Controls the computational effort for reasoning models
+- Options: `low`, `medium`, `high`
+- Guidelines:
+  - `low`: Fast responses, basic reasoning (default for simple queries)
+  - `medium`: Balanced speed and reasoning depth
+  - `high`: Maximum reasoning effort, best for complex analytical questions
+- **Supported Models**:
+  - **Ollama**: `gpt-oss:20b`, `gpt-oss:120b`
+  - **OpenAI**: `o3`, `o3-mini`, `o4-mini`, `o1` (o-series reasoning models)
+- **Note**: Models without reasoning support will show a warning and proceed without reasoning parameters
+- **Example**: `--thinking-budget high` for complex analytical questions
+
+**📖 For detailed usage examples and implementation details, check out [Thinking Budget Documentation](THINKING_BUDGET_FEATURE.md)**
+
+**💡 Quick Examples:**
+```bash
+# OpenAI o-series reasoning model
+python apps/document_rag.py --query "What are the main techniques LEANN explores?" \
+  --index-dir hnswbuild --backend hnsw \
+  --llm openai --llm-model o3 --thinking-budget medium
+
+# Ollama reasoning model
+python apps/document_rag.py --query "What are the main techniques LEANN explores?" \
+  --index-dir hnswbuild --backend hnsw \
+  --llm ollama --llm-model gpt-oss:20b --thinking-budget high
+```
+
+### Graph Degree (HNSW/DiskANN)
+
+**`--graph-degree`**
+- Number of connections per node in the graph
+- Higher = better recall but more memory
+- HNSW: 16-32 (default: 32)
+- DiskANN: 32-128 (default: 64)
+
+
+## Performance Optimization Checklist
+
+### If Embedding is Too Slow
+
+1. **Switch to smaller model**:
+   ```bash
+   # From large model
+   --embedding-model Qwen/Qwen3-Embedding-0.6B
+   # To small model
+   --embedding-model sentence-transformers/all-MiniLM-L6-v2
+   ```
+
+2. **Limit dataset size for testing**:
+   ```bash
+   --max-items 1000  # Process first 1k items only
+   ```
+
+3. **Use MLX on Apple Silicon** (optional optimization):
+   ```bash
+   --embedding-mode mlx --embedding-model mlx-community/Qwen3-Embedding-0.6B-8bit
+   ```
+    MLX might not be the best choice, as we tested and found that it only offers 1.3x acceleration compared to HF, so maybe using ollama is a better choice for embedding generation
+
+4. **Use Ollama**
+   ```bash
+   --embedding-mode ollama --embedding-model nomic-embed-text
+   ```
+   To discover additional embedding models in ollama, check out https://ollama.com/search?c=embedding or read more about embedding models at https://ollama.com/blog/embedding-models, please do check the model size that works best for you
+### If Search Quality is Poor
+
+1. **Increase retrieval count**:
+   ```bash
+   --top-k 30  # Retrieve more candidates
+   ```
+
+2. **Upgrade embedding model**:
+   ```bash
+   # For English
+   --embedding-model BAAI/bge-base-en-v1.5
+   # For multilingual
+   --embedding-model intfloat/multilingual-e5-large
+   ```
+
+## Understanding the Trade-offs
+
+Every configuration choice involves trade-offs:
+
+| Factor | Small/Fast | Large/Quality |
+|--------|------------|---------------|
+| Embedding Model | `all-MiniLM-L6-v2` | `Qwen/Qwen3-Embedding-0.6B` |
+| Chunk Size | 512 tokens | 128 tokens |
+| Index Type | HNSW | DiskANN |
+| LLM | `qwen3:1.7b` | `gpt-4o` |
+
+The key is finding the right balance for your specific use case. Start small and simple, measure performance, then scale up only where needed.
+
+## Low-resource setups
+
+If you don’t have a local GPU or builds/searches are too slow, use one or more of the options below.
+
+### 1) Use OpenAI embeddings (no local compute)
+
+Fastest path with zero local GPU requirements. Set your API key and use OpenAI embeddings during build and search:
+
+```bash
+export OPENAI_API_KEY=sk-...
+
+# Build with OpenAI embeddings
+leann build my-index \
+  --embedding-mode openai \
+  --embedding-model text-embedding-3-small
+
+# Search with OpenAI embeddings (recompute at query time)
+leann search my-index "your query" \
+  --recompute
+```
+
+### 2) Run remote builds with SkyPilot (cloud GPU)
+
+Offload embedding generation and index building to a GPU VM using [SkyPilot](https://docs.skypilot.co/en/latest/docs/index.html). A template is provided at `sky/leann-build.yaml`.
+
+```bash
+# One-time: install and configure SkyPilot
+pip install skypilot
+
+# Launch with defaults (L4:1) and mount ./data to ~/leann-data; the build runs automatically
+sky launch -c leann-gpu sky/leann-build.yaml
+
+# Override parameters via -e key=value (optional)
+sky launch -c leann-gpu sky/leann-build.yaml \
+  -e index_name=my-index \
+  -e backend=hnsw \
+  -e embedding_mode=sentence-transformers \
+  -e embedding_model=Qwen/Qwen3-Embedding-0.6B
+
+# Copy the built index back to your local .leann (use rsync)
+rsync -Pavz leann-gpu:~/.leann/indexes/my-index ./.leann/indexes/
+```
+
+### 3) Disable recomputation to trade storage for speed
+
+If you need lower latency and have more storage/memory, disable recomputation. This stores full embeddings and avoids recomputing at search time.
+
+```bash
+# Build without recomputation (HNSW requires non-compact in this mode)
+leann build my-index --no-recompute --no-compact
+
+# Search without recomputation
+leann search my-index "your query" --no-recompute
+```
+
+When to use:
+- Extreme low latency requirements (high QPS, interactive assistants)
+- Read-heavy workloads where storage is cheaper than latency
+- No always-available GPU
+
+Constraints:
+- HNSW: when `--no-recompute` is set, LEANN automatically disables compact mode during build
+- DiskANN: supported; `--no-recompute` skips selective recompute during search
+
+Storage impact:
+- Storing N embeddings of dimension D with float32 requires approximately N × D × 4 bytes
+- Example: 1,000,000 chunks × 768 dims × 4 bytes ≈ 2.86 GB (plus graph/metadata)
+
+Converting an existing index (rebuild required):
+```bash
+# Rebuild in-place (ensure you still have original docs or can regenerate chunks)
+leann build my-index --force --no-recompute --no-compact
+```
+
+Python API usage:
+```python
+from leann import LeannSearcher
+
+searcher = LeannSearcher("/path/to/my-index.leann")
+results = searcher.search("your query", top_k=10, recompute_embeddings=False)
+```
+
+Trade-offs:
+- Lower latency and fewer network hops at query time
+- Significantly higher storage (10–100× vs selective recomputation)
+- Slightly larger memory footprint during build and search
+
+Quick benchmark results (`benchmarks/benchmark_no_recompute.py` with 5k texts, complexity=32):
+
+- HNSW
+
+  ```text
+  recompute=True:  search_time=0.818s, size=1.1MB
+  recompute=False: search_time=0.012s, size=16.6MB
+  ```
+
+- DiskANN
+
+  ```text
+  recompute=True:  search_time=0.041s, size=5.9MB
+  recompute=False: search_time=0.013s, size=24.6MB
+  ```
+
+Conclusion:
+- **HNSW**: `no-recompute` is significantly faster (no embedding recomputation) but requires much more storage (stores all embeddings)
+- **DiskANN**: `no-recompute` uses PQ + partial real distances during traversal (slower but higher accuracy), while `recompute=True` uses pure PQ traversal + final reranking (faster traversal, enables build-time partitioning for smaller storage)
+
+
+
+## Further Reading
+
+- [Lessons Learned Developing LEANN](https://yichuan-w.github.io/blog/lessons_learned_in_dev_leann/)
+- [LEANN Technical Paper](https://arxiv.org/abs/2506.08276)
+- [DiskANN Original Paper](https://suhasjs.github.io/files/diskann_neurips19.pdf)
+- [SSD-based Graph Partitioning](https://github.com/SonglinLife/SSD_BASED_PLAN)

docs/faq.md

@@ -0,0 +1,58 @@
+# FAQ
+
+## 1. My building time seems long
+
+You can speed up the process by using a lightweight embedding model. Add this to your arguments:
+
+```bash
+--embedding-model sentence-transformers/all-MiniLM-L6-v2
+```
+**Model sizes:** `all-MiniLM-L6-v2` (30M parameters), `facebook/contriever` (~100M parameters), `Qwen3-0.6B` (600M parameters)
+
+## 2. When should I use prompt templates?
+
+**Use prompt templates ONLY with task-specific embedding models** like Google's EmbeddingGemma. These models are specially trained to use different prompts for documents vs queries.
+
+**DO NOT use with regular models** like `nomic-embed-text`, `text-embedding-3-small`, or `bge-base-en-v1.5` - adding prompts to these models will corrupt the embeddings.
+
+**Example usage with EmbeddingGemma:**
+```bash
+# Build with document prompt
+leann build my-docs --embedding-prompt-template "title: none | text: "
+
+# Search with query prompt
+leann search my-docs --query "your question" --embedding-prompt-template "task: search result | query: "
+```
+
+See the [Configuration Guide: Task-Specific Prompt Templates](configuration-guide.md#task-specific-prompt-templates) for detailed usage.
+
+## 3. Why is LM Studio loading multiple copies of my model?
+
+This was fixed in recent versions. LEANN now properly unloads models after querying metadata, respecting your LM Studio JIT auto-evict settings.
+
+**If you still see duplicates:**
+- Update to the latest LEANN version
+- Restart LM Studio to clear loaded models
+- Check that you have JIT auto-evict enabled in LM Studio settings
+
+**How it works now:**
+1. LEANN loads model temporarily to get context length
+2. Immediately unloads after query
+3. LM Studio JIT loads model on-demand for actual embeddings
+4. Auto-evicts per your settings
+
+## 4. Do I need Node.js and @lmstudio/sdk?
+
+**No, it's completely optional.** LEANN works perfectly fine without them using a built-in token limit registry.
+
+**Benefits if you install it:**
+- Automatic context length detection for LM Studio models
+- No manual registry maintenance
+- Always gets accurate token limits from the model itself
+
+**To install (optional):**
+```bash
+npm install -g @lmstudio/sdk
+```
+
+See [Configuration Guide: LM Studio Auto-Detection](configuration-guide.md#lm-studio-auto-detection-optional) for details.

docs/features.md

@@ -0,0 +1,23 @@
+# ✨ Detailed Features
+
+## 🔥 Core Features
+
+- **🔄 Real-time Embeddings** - Eliminate heavy embedding storage with dynamic computation using optimized ZMQ servers and highly optimized search paradigm (overlapping and batching) with highly optimized embedding engine
+- **🧠 AST-Aware Code Chunking** - Intelligent code chunking that preserves semantic boundaries (functions, classes, methods) for Python, Java, C#, and TypeScript files
+- **📈 Scalable Architecture** - Handles millions of documents on consumer hardware; the larger your dataset, the more LEANN can save
+- **🎯 Graph Pruning** - Advanced techniques to minimize the storage overhead of vector search to a limited footprint
+- **🏗️ Pluggable Backends** - HNSW/FAISS (default), with optional DiskANN for large-scale deployments
+
+## 🛠️ Technical Highlights
+- **🔄 Recompute Mode** - Highest accuracy scenarios while eliminating vector storage overhead
+- **⚡ Zero-copy Operations** - Minimize IPC overhead by transferring distances instead of embeddings
+- **🚀 High-throughput Embedding Pipeline** - Optimized batched processing for maximum efficiency
+- **🎯 Two-level Search** - Novel coarse-to-fine search overlap for accelerated query processing (optional)
+- **💾 Memory-mapped Indices** - Fast startup with raw text mapping to reduce memory overhead
+- **🚀 MLX Support** - Ultra-fast recompute/build with quantized embedding models, accelerating building and search ([minimal example](../examples/mlx_demo.py))
+
+## 🎨 Developer Experience
+
+- **Simple Python API** - Get started in minutes
+- **Extensible backend system** - Easy to add new algorithms
+- **Comprehensive examples** - From basic usage to production deployment

docs/flashlib_backend_guide.md

@@ -0,0 +1,179 @@
+# FlashLib Backend Guide
+
+LEANN now ships an optional, GPU-accelerated search backend powered by
+[**FlashLib**](https://github.com/FlashML-org/flashlib) — a library of classical
+machine-learning operators built on Triton and CuteDSL. This guide covers what it
+is, when to use it, how to install it, and how it plugs into LEANN.
+
+## What is FlashLib?
+
+FlashLib (`pip install flashlib`) is a GPU library of classical ML primitives
+(k-means, DBSCAN, PCA, SVD, UMAP, t-SNE, IVF-Flat ANN, and more). LEANN uses its
+`IVFFlat` index — an inverted-file flat approximate-nearest-neighbor (ANN) index
+that runs entirely on CUDA tensors. At a fixed `(nlist, nprobe)` it probes the same
+candidate set as a reference IVF-Flat (FAISS / cuVS), so recall is predictable,
+while the search itself is accelerated on the GPU.
+
+```python
+import torch
+from flashlib import IVFFlat
+
+db = torch.randn(1_000_000, 128, device="cuda")
+index = IVFFlat(nlist=1024, nprobe=16).fit(db)
+distances, indices = index.kneighbors(torch.randn(10_000, 128, device="cuda"), n_neighbors=10)
+```
+
+## When to use it
+
+| Backend | Best for | Storage | Hardware |
+|---------|----------|---------|----------|
+| `hnsw` (default) | Laptop / CPU, max storage savings via recomputation | ~3% of raw (pruned graph) | CPU |
+| `diskann` | Larger-than-memory datasets | On-disk graph | CPU |
+| `ivf` | Incremental add/remove without rebuild | Full vectors (FAISS) | CPU |
+| **`flashlib`** | **High-throughput search on a CUDA GPU** | Full vectors (`.npy`) | **CUDA GPU** |
+| **`flashlib_ivf`** | **GPU IVF-Flat (approximate) — the GPU counterpart of `ivf`** | Full vectors (`.pt`) | **CUDA GPU** |
+
+Use FlashLib when you already have a GPU and want fast IVF-Flat search. It stores
+the full float32 vectors rather than a pruned graph, so it trades LEANN's storage
+savings for raw GPU search speed.
+
+## Requirements
+
+- A **CUDA GPU** (required at *search* time; building the index only needs numpy).
+- `flashlib` and `torch` (installed automatically with the extra below).
+
+## Installation
+
+The backend is **optional** — it is not pulled in by a default LEANN install.
+
+```bash
+# From a LEANN source checkout
+uv sync --extra flashlib
+
+# Or as a standalone package
+pip install leann-backend-flashlib
+```
+
+LEANN auto-discovers any installed `leann-backend-*` package, so once it is
+installed the `flashlib` backend name is available with no further configuration.
+
+## Usage
+
+### Python API
+
+```python
+from leann import LeannBuilder, LeannSearcher
+
+builder = LeannBuilder(backend_name="flashlib")   # nlist=1024, distance_metric="mips"
+builder.add_text("LEANN recomputes embeddings on the fly to cut storage by ~97%.")
+builder.add_text("FlashLib runs IVF-Flat search on the GPU.")
+builder.build_index("demo.leann")
+
+searcher = LeannSearcher("demo.leann")
+results = searcher.search("How does LEANN save storage?", top_k=3)
+for r in results:
+    print(r.score, r.text)
+```
+
+### Example apps / CLI
+
+```bash
+source .venv/bin/activate
+python -m apps.document_rag \
+    --query "What are the main techniques LEANN explores?" \
+    --backend-name flashlib
+```
+
+## How it works
+
+FlashLib's `IVFFlat` builds its index in GPU memory and has **no on-disk format**.
+The LEANN backend bridges that gap:
+
+1. **Build** (`FlashlibBuilder`): persists the raw float32 vectors as
+   `<index>.flashlib.npy` and an id map as `<index>.flashlib_id_map.json`.
+2. **Search** (`FlashlibSearcher`): loads the vectors into a CUDA tensor and
+   reconstructs the index once via `IVFFlat(nlist, nprobe).fit(db)` at start-up,
+   then answers every query with `index.kneighbors(...)`.
+
+FlashLib's only distance metric is **squared L2**. For `mips` / `cosine` the
+backend L2-normalizes both the database and the query vectors; on unit vectors,
+squared-L2 ranking is equivalent to inner-product / cosine ranking, so results
+match the other backends. `nlist` is clamped to the corpus size (the k-means
+constraint), and `nprobe` is derived from the search `complexity` knob.
+
+## Parameters
+
+| Parameter | Default | Meaning |
+|-----------|---------|---------|
+| `nlist` (build) | `1024` | Number of IVF partitions; clamped to the number of vectors. |
+| `distance_metric` (build) | `"mips"` | `mips`, `cosine`, or `l2`. |
+| `nprobe` (search) | derived from `complexity` | Partitions probed per query — the recall knob (higher = better recall, slower). |
+
+## FlashLib IVF backend (`flashlib_ivf`)
+
+The **`flashlib_ivf`** backend (`leann-backend-flashlib-ivf`) is the GPU counterpart
+of LEANN's FAISS `ivf` backend: an IVF-Flat (inverted file) index that
+coarse-quantizes the corpus into `nlist` cells with k-means and, at search time,
+scans only the `nprobe` nearest cells — entirely on CUDA tensors via FlashLib's
+`flash_ivf_flat`. At a fixed `(nlist, nprobe)` the GPU and CPU IVF probe nearly the
+same candidate set, so recall is comparable; the only difference is GPU vs CPU
+kernels. `nprobe` is the recall knob (defaults to `min(complexity, nlist)`).
+
+```bash
+uv sync --extra flashlib-ivf          # or: pip install leann-backend-flashlib-ivf
+```
+
+```python
+from leann import LeannBuilder, LeannSearcher
+
+builder = LeannBuilder(backend_name="flashlib_ivf", nlist=4096, distance_metric="cosine")
+builder.add_text("LEANN recomputes embeddings on the fly to cut storage by ~97%.")
+builder.build_index("demo.leann")
+
+searcher = LeannSearcher("demo.leann")
+results = searcher.search("How does LEANN save storage?", top_k=10, complexity=32)  # nprobe=32
+```
+
+**How it works:** the builder trains the coarse quantizer on the GPU and persists the
+built index tensors with `torch.save` (`<index>.flashlib_ivf.pt`) plus an id map
+(`<index>.flashlib_ivf_id_map.json`); the searcher reloads them onto the GPU once (no
+k-means re-train). A CUDA GPU is required at **both** build (k-means) and search time.
+FlashLib IVF ranks by squared L2, so `mips`/`cosine` L2-normalize the vectors (squared-L2
+ranking then matches inner-product/cosine).
+
+### Speed — IVF (GPU) vs IVF (CPU)
+
+```bash
+python benchmarks/flashlib_ivf_vs_faiss_ivf.py \
+    --sizes 100000 1000000 --nprobe-sweep 1 8 32 128 --cpu-threads 8
+```
+
+`flashlib_ivf` (GPU) vs the FAISS `ivf` backend (CPU) at the **same `nlist`**, sweeping
+`nprobe` (NVIDIA H200, `faiss-cpu` at 8 threads, 768-dim, top-k=10). GPU latency stays
+~flat while CPU latency grows linearly with `nprobe`, so the GPU lead widens exactly as
+you raise recall:
+
+| Corpus | nprobe | GPU lat | CPU lat | GPU q/s | CPU q/s | Recall (GPU/CPU) | Speedup (lat / tpt) |
+|--------|--------|---------|---------|---------|---------|------------------|---------------------|
+| 1M | 8 | 0.45 ms | 1.14 ms | 107k | 5.9k | 0.340 / 0.321 | 2.6× / 18× |
+| 1M | 32 | 0.46 ms | 3.00 ms | 141k | 1.9k | 0.400 / 0.350 | 6.5× / 75× |
+| 1M | 128 | 0.55 ms | 9.91 ms | 95k | 0.6k | 0.539 / 0.423 | 18× / 159× |
+
+Build (1M, `nlist=4096`): GPU **10.6 s** vs FAISS CPU **140.7 s** — a **13× faster
+build** (GPU k-means vs CPU k-means training).
+
+Honest caveats: at very low `nprobe` (e.g. 1) single-query GPU latency (~0.44 ms) is
+*higher* than CPU, because the per-query work is tiny and GPU kernel-launch overhead
+dominates; the GPU advantage grows with `nprobe` (higher recall), batch size, and corpus
+size. The absolute recall above is low because the synthetic mixture-of-Gaussians corpus
+has more clusters than `nlist`; on real embeddings recall is far higher — this benchmark
+isolates the GPU-vs-CPU *relative* comparison at matched `(nlist, nprobe)`.
+
+## Notes & limitations
+
+- GPU-only at search time; building an index works on a CPU-only machine.
+- Stores full vectors, so it does not benefit from LEANN's graph-pruning storage
+  savings — pick `hnsw` if minimizing disk footprint is the priority.
+- Query embeddings are computed through the standard LEANN embedding path (the
+  HNSW ZMQ embedding server when available, otherwise direct model loading), so
+  any LEANN-supported embedding model works.

docs/grep_search.md

@@ -0,0 +1,149 @@
+# LEANN Grep Search Usage Guide
+
+## Overview
+
+LEANN's grep search functionality provides exact text matching for finding specific code patterns, error messages, function names, or exact phrases in your indexed documents.
+
+## Basic Usage
+
+### Simple Grep Search
+
+```python
+from leann.api import LeannSearcher
+
+searcher = LeannSearcher("your_index_path")
+
+# Exact text search
+results = searcher.search("def authenticate_user", use_grep=True, top_k=5)
+
+for result in results:
+    print(f"Score: {result.score}")
+    print(f"Text: {result.text[:100]}...")
+    print("-" * 40)
+```
+
+### Comparison: Semantic vs Grep Search
+
+```python
+# Semantic search - finds conceptually similar content
+semantic_results = searcher.search("machine learning algorithms", top_k=3)
+
+# Grep search - finds exact text matches
+grep_results = searcher.search("def train_model", use_grep=True, top_k=3)
+```
+
+## When to Use Grep Search
+
+### Use Cases
+
+- **Code Search**: Finding specific function definitions, class names, or variable references
+- **Error Debugging**: Locating exact error messages or stack traces
+- **Documentation**: Finding specific API endpoints or exact terminology
+
+### Examples
+
+```python
+# Find function definitions
+functions = searcher.search("def __init__", use_grep=True)
+
+# Find import statements
+imports = searcher.search("from sklearn import", use_grep=True)
+
+# Find specific error types
+errors = searcher.search("FileNotFoundError", use_grep=True)
+
+# Find TODO comments
+todos = searcher.search("TODO:", use_grep=True)
+
+# Find configuration entries
+configs = searcher.search("server_port=", use_grep=True)
+```
+
+## Technical Details
+
+### How It Works
+
+1. **File Location**: Grep search operates on the raw text stored in `.jsonl` files
+2. **Command Execution**: Uses the system `grep` command with case-insensitive search
+3. **Result Processing**: Parses JSON lines and extracts text and metadata
+4. **Scoring**: Simple frequency-based scoring based on query term occurrences
+
+### Search Process
+
+```
+Query: "def train_model"
+  ↓
+grep -i -n "def train_model" documents.leann.passages.jsonl
+  ↓
+Parse matching JSON lines
+  ↓
+Calculate scores based on term frequency
+  ↓
+Return top_k results
+```
+
+### Scoring Algorithm
+
+```python
+# Term frequency in document
+score = text.lower().count(query.lower())
+```
+
+Results are ranked by score (highest first), with higher scores indicating more occurrences of the search term.
+
+## Error Handling
+
+### Common Issues
+
+#### Grep Command Not Found
+```
+RuntimeError: grep command not found. Please install grep or use semantic search.
+```
+
+**Solution**: Install grep on your system:
+- **Ubuntu/Debian**: `sudo apt-get install grep`
+- **macOS**: grep is pre-installed
+- **Windows**: Use WSL or install grep via Git Bash/MSYS2
+
+#### No Results Found
+```python
+# Check if your query exists in the raw data
+results = searcher.search("your_query", use_grep=True)
+if not results:
+    print("No exact matches found. Try:")
+    print("1. Check spelling and case")
+    print("2. Use partial terms")
+    print("3. Switch to semantic search")
+```
+
+## Complete Example
+
+```python
+#!/usr/bin/env python3
+"""
+Grep Search Example
+Demonstrates grep search for exact text matching.
+"""
+
+from leann.api import LeannSearcher
+
+def demonstrate_grep_search():
+    # Initialize searcher
+    searcher = LeannSearcher("my_index")
+
+    print("=== Function Search ===")
+    functions = searcher.search("def __init__", use_grep=True, top_k=5)
+    for i, result in enumerate(functions, 1):
+        print(f"{i}. Score: {result.score}")
+        print(f"   Preview: {result.text[:60]}...")
+        print()
+
+    print("=== Error Search ===")
+    errors = searcher.search("FileNotFoundError", use_grep=True, top_k=3)
+    for result in errors:
+        print(f"Content: {result.text.strip()}")
+        print("-" * 40)
+
+if __name__ == "__main__":
+    demonstrate_grep_search()
+```

docs/issue-proposals/smart-embedding-default.md

@@ -0,0 +1,41 @@
+# Smart default embedding model based on platform and corpus size
+
+## Summary
+
+Propose platform- and corpus-aware default embedding model selection for `leann build` when `--embedding-model` is not explicitly specified. This would improve out-of-the-box experience for different deployment scenarios (macOS CPU, NVIDIA GPU, etc.) without changing behavior when users pass an explicit model.
+
+## Motivation
+
+- **Current default**: `facebook/contriever` (~420MB, 768 dim) — heavy for CPU-only builds on large corpora
+- **macOS users** often hit slow builds on 20K+ chunks; lighter models like `all-MiniLM-L6-v2` (~90MB) are much faster
+- **NVIDIA GPU users** can leverage stronger models; smaller corpora benefit from quality (e.g. Qwen3-Embedding-0.6B), larger ones from balanced models (e.g. bge-base-en-v1.5)
+
+## Proposed logic
+
+| Platform | Chunk count | Default model |
+|----------|-------------|---------------|
+| **macOS** | ≥ 20,000 | `sentence-transformers/all-MiniLM-L6-v2` |
+| **macOS** | < 20,000 | `intfloat/e5-small-v2` |
+| **NVIDIA GPU** | < 5,000 | `Qwen/Qwen3-Embedding-0.6B` |
+| **NVIDIA GPU** | ≥ 5,000 | `BAAI/bge-base-en-v1.5` |
+| **Other** | any | `facebook/contriever` (unchanged) |
+
+## Implementation notes
+
+1. **Platform detection**: `torch.cuda.is_available()` for NVIDIA; `sys.platform == "darwin"` for macOS
+2. **Chunk count**: Known only after loading/chunking; may need to either:
+   - Do a lightweight pre-scan (e.g. file count × rough chunks per file), or
+   - Defer default choice until after first chunking pass (and cache for incremental)
+3. **Explicit override**: If user passes `--embedding-model`, always use it; this logic applies only when the flag is omitted
+
+## Model references
+
+- `sentence-transformers/all-MiniLM-L6-v2`: ~90MB, 384 dim, fast on CPU
+- `intfloat/e5-small-v2`: ~90MB, 384 dim
+- `Qwen/Qwen3-Embedding-0.6B`: 0.6B params, 1024 dim, strong retrieval
+- `BAAI/bge-base-en-v1.5`: ~110M params, 768 dim, good MTEB scores
+
+## Open questions
+
+- Should we add a `--embedding-model auto` to explicitly opt into this logic?
+- Pre-scan vs post-chunk decision: trade-off between accuracy and implementation complexity

docs/metadata_filtering.md

@@ -0,0 +1,300 @@
+# LEANN Metadata Filtering Usage Guide
+
+## Overview
+
+Leann possesses metadata filtering capabilities that allow you to filter search results based on arbitrary metadata fields set during chunking. This feature enables use cases like spoiler-free book search, document filtering by date/type, code search by file type, and potentially much more.
+
+## Basic Usage
+
+### Adding Metadata to Your Documents
+
+When building your index, add metadata to each text chunk:
+
+```python
+from leann.api import LeannBuilder
+
+builder = LeannBuilder("hnsw")
+
+# Add text with metadata
+builder.add_text(
+    text="Chapter 1: Alice falls down the rabbit hole",
+    metadata={
+        "chapter": 1,
+        "character": "Alice",
+        "themes": ["adventure", "curiosity"],
+        "word_count": 150
+    }
+)
+
+builder.build_index("alice_in_wonderland_index")
+```
+
+### Searching with Metadata Filters
+
+Use the `metadata_filters` parameter in search calls:
+
+```python
+from leann.api import LeannSearcher
+
+searcher = LeannSearcher("alice_in_wonderland_index")
+
+# Search with filters
+results = searcher.search(
+    query="What happens to Alice?",
+    top_k=10,
+    metadata_filters={
+        "chapter": {"<=": 5},           # Only chapters 1-5
+        "spoiler_level": {"!=": "high"} # No high spoilers
+    }
+)
+```
+
+## Filter Syntax
+
+### Basic Structure
+
+```python
+metadata_filters = {
+    "field_name": {"operator": value},
+    "another_field": {"operator": value}
+}
+```
+
+### Supported Operators
+
+#### Comparison Operators
+- `"=="`: Equal to
+- `"!="`: Not equal to
+- `"<"`: Less than
+- `"<="`: Less than or equal
+- `">"`: Greater than
+- `">="`: Greater than or equal
+
+```python
+# Examples
+{"chapter": {"==": 1}}           # Exactly chapter 1
+{"page": {">": 100}}            # Pages after 100
+{"rating": {">=": 4.0}}         # Rating 4.0 or higher
+{"word_count": {"<": 500}}      # Short passages
+```
+
+#### Membership Operators
+- `"in"`: Value is in list
+- `"not_in"`: Value is not in list
+
+```python
+# Examples
+{"character": {"in": ["Alice", "Bob"]}}      # Alice OR Bob
+{"genre": {"not_in": ["horror", "thriller"]}} # Exclude genres
+{"tags": {"in": ["fiction", "adventure"]}}   # Any of these tags
+```
+
+#### String Operators
+- `"contains"`: String contains substring
+- `"starts_with"`: String starts with prefix
+- `"ends_with"`: String ends with suffix
+
+```python
+# Examples
+{"title": {"contains": "alice"}}        # Title contains "alice"
+{"filename": {"ends_with": ".py"}}      # Python files
+{"author": {"starts_with": "Dr."}}      # Authors with "Dr." prefix
+```
+
+#### Boolean Operators
+- `"is_true"`: Field is truthy
+- `"is_false"`: Field is falsy
+
+```python
+# Examples
+{"is_published": {"is_true": True}}     # Published content
+{"is_draft": {"is_false": False}}       # Not drafts
+```
+
+### Multiple Operators on Same Field
+
+You can apply multiple operators to the same field (AND logic):
+
+```python
+metadata_filters = {
+    "word_count": {
+        ">=": 100,    # At least 100 words
+        "<=": 500     # At most 500 words
+    }
+}
+```
+
+### Compound Filters
+
+Multiple fields are combined with AND logic:
+
+```python
+metadata_filters = {
+    "chapter": {"<=": 10},              # Up to chapter 10
+    "character": {"==": "Alice"},       # About Alice
+    "spoiler_level": {"!=": "high"}     # No major spoilers
+}
+```
+
+## Use Case Examples
+
+### 1. Spoiler-Free Book Search
+
+```python
+# Reader has only read up to chapter 5
+def search_spoiler_free(query, max_chapter):
+    return searcher.search(
+        query=query,
+        metadata_filters={
+            "chapter": {"<=": max_chapter},
+            "spoiler_level": {"in": ["none", "low"]}
+        }
+    )
+
+results = search_spoiler_free("What happens to Alice?", max_chapter=5)
+```
+
+### 2. Document Management by Date
+
+```python
+# Find recent documents
+recent_docs = searcher.search(
+    query="project updates",
+    metadata_filters={
+        "date": {">=": "2024-01-01"},
+        "document_type": {"==": "report"}
+    }
+)
+```
+
+### 3. Code Search by File Type
+
+```python
+# Search only Python files
+python_code = searcher.search(
+    query="authentication function",
+    metadata_filters={
+        "file_extension": {"==": ".py"},
+        "lines_of_code": {"<": 100}
+    }
+)
+```
+
+### 4. Content Filtering by Audience
+
+```python
+# Age-appropriate content
+family_content = searcher.search(
+    query="adventure stories",
+    metadata_filters={
+        "age_rating": {"in": ["G", "PG"]},
+        "content_warnings": {"not_in": ["violence", "adult_themes"]}
+    }
+)
+```
+
+### 5. Multi-Book Series Management
+
+```python
+# Search across first 3 books only
+early_series = searcher.search(
+    query="character development",
+    metadata_filters={
+        "series": {"==": "Harry Potter"},
+        "book_number": {"<=": 3}
+    }
+)
+```
+
+## Running the Example
+
+You can see metadata filtering in action with our spoiler-free book RAG example:
+
+```bash
+# Don't forget to set up the environment
+uv venv
+source .venv/bin/activate
+
+# Set your OpenAI API key (required for embeddings, but you can update the example locally and use ollama instead)
+export OPENAI_API_KEY="your-api-key-here"
+
+# Run the spoiler-free book RAG example
+uv run examples/spoiler_free_book_rag.py
+```
+
+This example demonstrates:
+- Building an index with metadata (chapter numbers, characters, themes, locations)
+- Searching with filters to avoid spoilers (e.g., only show results up to chapter 5)
+- Different scenarios for readers at various points in the book
+
+The example uses Alice's Adventures in Wonderland as sample data and shows how you can search for information without revealing plot points from later chapters.
+
+## Advanced Patterns
+
+### Custom Chunking with metadata
+
+```python
+def chunk_book_with_metadata(book_text, book_info):
+    chunks = []
+
+    for chapter_num, chapter_text in parse_chapters(book_text):
+        # Extract entities, themes, etc.
+        characters = extract_characters(chapter_text)
+        themes = classify_themes(chapter_text)
+        spoiler_level = assess_spoiler_level(chapter_text, chapter_num)
+
+        # Create chunks with rich metadata
+        for paragraph in split_paragraphs(chapter_text):
+            chunks.append({
+                "text": paragraph,
+                "metadata": {
+                    "book_title": book_info["title"],
+                    "chapter": chapter_num,
+                    "characters": characters,
+                    "themes": themes,
+                    "spoiler_level": spoiler_level,
+                    "word_count": len(paragraph.split()),
+                    "reading_level": calculate_reading_level(paragraph)
+                }
+            })
+
+    return chunks
+```
+
+## Performance Considerations
+
+### Efficient Filtering Strategies
+
+1. **Post-search filtering**: Applies filters after vector search, which should be efficient for typical result sets (10-100 results).
+
+2. **Metadata design**: Keep metadata fields simple and avoid deeply nested structures.
+
+### Best Practices
+
+1. **Consistent metadata schema**: Use consistent field names and value types across your documents.
+
+2. **Reasonable metadata size**: Keep metadata reasonably sized to avoid storage overhead.
+
+3. **Type consistency**: Use consistent data types for the same fields (e.g., always integers for chapter numbers).
+
+4. **Index multiple granularities**: Consider chunking at different levels (paragraph, section, chapter) with appropriate metadata.
+
+### Adding Metadata to Existing Indices
+
+To add metadata filtering to existing indices, you'll need to rebuild them with metadata:
+
+```python
+# Read existing passages and add metadata
+def add_metadata_to_existing_chunks(chunks):
+    for chunk in chunks:
+        # Extract or assign metadata based on content
+        chunk["metadata"] = extract_metadata(chunk["text"])
+    return chunks
+
+# Rebuild index with metadata
+enhanced_chunks = add_metadata_to_existing_chunks(existing_chunks)
+builder = LeannBuilder("hnsw")
+for chunk in enhanced_chunks:
+    builder.add_text(chunk["text"], chunk["metadata"])
+builder.build_index("enhanced_index")
+```

docs/normalized_embeddings.md

@@ -0,0 +1,75 @@
+# Normalized Embeddings Support in LEANN
+
+LEANN now automatically detects normalized embedding models and sets the appropriate distance metric for optimal performance.
+
+## What are Normalized Embeddings?
+
+Normalized embeddings are vectors with L2 norm = 1 (unit vectors). These embeddings are optimized for cosine similarity rather than Maximum Inner Product Search (MIPS).
+
+## Automatic Detection
+
+When you create a `LeannBuilder` instance with a normalized embedding model, LEANN will:
+
+1. **Automatically set `distance_metric="cosine"`** if not specified
+2. **Show a warning** if you manually specify a different distance metric
+3. **Provide optimal search performance** with the correct metric
+
+## Supported Normalized Embedding Models
+
+### OpenAI
+All OpenAI text embedding models are normalized:
+- `text-embedding-ada-002`
+- `text-embedding-3-small`
+- `text-embedding-3-large`
+
+### Voyage AI
+All Voyage AI embedding models are normalized:
+- `voyage-2`
+- `voyage-3`
+- `voyage-large-2`
+- `voyage-multilingual-2`
+- `voyage-code-2`
+
+### Cohere
+All Cohere embedding models are normalized:
+- `embed-english-v3.0`
+- `embed-multilingual-v3.0`
+- `embed-english-light-v3.0`
+- `embed-multilingual-light-v3.0`
+
+## Example Usage
+
+```python
+from leann.api import LeannBuilder
+
+# Automatic detection - will use cosine distance
+builder = LeannBuilder(
+    backend_name="hnsw",
+    embedding_model="text-embedding-3-small",
+    embedding_mode="openai"
+)
+# Warning: Detected normalized embeddings model 'text-embedding-3-small'...
+# Automatically setting distance_metric='cosine'
+
+# Manual override (not recommended)
+builder = LeannBuilder(
+    backend_name="hnsw",
+    embedding_model="text-embedding-3-small",
+    embedding_mode="openai",
+    distance_metric="mips"  # Will show warning
+)
+# Warning: Using 'mips' distance metric with normalized embeddings...
+```
+
+## Non-Normalized Embeddings
+
+Models like `facebook/contriever` and other sentence-transformers models that are not normalized will continue to use MIPS by default, which is optimal for them.
+
+## Why This Matters
+
+Using the wrong distance metric with normalized embeddings can lead to:
+- **Poor search quality** due to HNSW's early termination with narrow score ranges
+- **Incorrect ranking** of search results
+- **Suboptimal performance** compared to using the correct metric
+
+For more details on why this happens, see our analysis in the [embedding detection code](../packages/leann-core/src/leann/api.py) which automatically handles normalized embeddings and MIPS distance metric issues.

docs/openclaw-setup.md

@@ -0,0 +1,166 @@
+# OpenClaw + LEANN Setup Guide
+
+Two ways to connect LEANN to your OpenClaw agent: **MCP server** (recommended)
+or **ClawHub skill**.
+
+---
+
+## Option A: MCP Server (Recommended)
+
+OpenClaw natively supports MCP tools. LEANN ships an MCP server that exposes
+`leann_search` and `leann_list` as tools your agent can call directly.
+
+### 1. Install LEANN
+
+```bash
+pip install leann-core
+# or
+uv tool install leann-core --with leann
+```
+
+### 2. Build an index on your memory files
+
+Using Ollama embeddings (recommended if you already run Ollama):
+
+```bash
+leann build openclaw-memory \
+  --docs ~/.openclaw/workspace/MEMORY.md ~/.openclaw/workspace/memory/ \
+  --embedding-mode ollama \
+  --embedding-model nomic-embed-text
+```
+
+Or using local sentence-transformers (no Ollama required):
+
+```bash
+leann build openclaw-memory \
+  --docs ~/.openclaw/workspace/MEMORY.md ~/.openclaw/workspace/memory/ \
+  --embedding-mode sentence-transformers \
+  --embedding-model all-MiniLM-L6-v2
+```
+
+Add extra directories if you have them:
+
+```bash
+leann build openclaw-memory \
+  --docs ~/.openclaw/workspace/MEMORY.md \
+        ~/.openclaw/workspace/memory/ \
+        ~/Documents/notes/ \
+  --embedding-mode ollama \
+  --embedding-model nomic-embed-text
+```
+
+### 3. Register the MCP server with OpenClaw
+
+Add to `~/.openclaw/openclaw.json`:
+
+```json5
+{
+  // ... your existing config ...
+  "mcpServers": {
+    "leann": {
+      "command": "leann_mcp",
+      "args": [],
+      "env": {}
+    }
+  }
+}
+```
+
+### 4. Use it
+
+Ask your agent:
+- "Search my memories for database decisions"
+- "What did we decide about the API design?"
+- "Find my notes on deployment"
+
+The agent will call `leann_search` via MCP and return structured results.
+
+### 5. Keep the index fresh
+
+```bash
+# Re-run build (idempotent — only processes changed files)
+leann build openclaw-memory \
+  --docs ~/.openclaw/workspace/MEMORY.md ~/.openclaw/workspace/memory/
+
+# Or use watch mode for continuous auto-sync
+leann watch openclaw-memory --interval 30
+```
+
+---
+
+## Option B: ClawHub Skill
+
+If you prefer the skill-based approach:
+
+```bash
+clawhub install leann-team/leann-memory
+```
+
+Or copy `skills/leann-memory/` from this repo to
+`~/.openclaw/workspace/skills/leann-memory/`.
+
+The skill tells your agent how to call `leann search` via shell commands.
+Setup steps (install + build index) are the same as above.
+
+---
+
+## Important: Ollama Configuration
+
+If you use Ollama as your OpenClaw model provider, make sure your
+`~/.openclaw/openclaw.json` uses the **native Ollama API** — not the
+OpenAI-compatible endpoint:
+
+```json5
+{
+  "models": {
+    "providers": {
+      "ollama": {
+        "baseUrl": "http://127.0.0.1:11434",  // no /v1 suffix
+        "apiKey": "ollama-local",
+        "api": "ollama"  // NOT "openai-completions" or "openai-responses"
+      }
+    }
+  }
+}
+```
+
+Using `"openai-completions"` or `"openai-responses"` silently breaks tool
+calling — the model outputs tool calls as plain text instead of structured
+`tool_calls`. See [astral-sh/ty#21243](https://github.com/openclaw/openclaw/issues/21243).
+
+---
+
+## Storage Comparison
+
+| Scenario | Default memory-core | LEANN |
+|---|---|---|
+| 1 year daily logs (~12K chunks) | ~23 MB | **~0.7 MB** |
+| + session transcripts (~100K chunks) | ~190 MB | **~6 MB** |
+| + 10 GB indexed documents (~500K chunks) | ~950 MB | **~30 MB** |
+
+All numbers assume 384-dimensional embeddings (all-MiniLM-L6-v2 or
+nomic-embed-text).
+
+---
+
+## Troubleshooting
+
+**"leann: command not found"**
+Ensure LEANN is on your PATH. If installed via `uv tool install`, run
+`uv tool update-shell` and restart your terminal.
+
+**"Index not found"**
+Run `leann list` to see available indexes. Build one first with `leann build`.
+
+**Slow first search**
+The first query loads the embedding model (~90 MB). Subsequent queries reuse the
+warm daemon and are fast (~0.5s). Use `leann warmup openclaw-memory` to
+pre-warm.
+
+**Memory files changed but search results are stale**
+Re-run `leann build openclaw-memory --docs ...` — it detects changes
+automatically and only re-indexes what changed.
+
+**Agent doesn't use LEANN tools**
+Make sure your Ollama model supports tool calling (e.g. `qwen3:8b` or larger).
+Smaller models like `qwen3:4b` may not reliably invoke tools.

docs/react_agent.md

@@ -0,0 +1,235 @@
+# LEANN ReAct Agent Guide
+
+## Overview
+
+The LEANN ReAct (Reasoning + Acting) Agent enables **multiturn retrieval and reasoning** for complex queries that require multiple search iterations. Unlike the standard `leann ask` command which performs a single search and answer, the ReAct agent can:
+
+- **Reason** about what information is needed
+- **Act** by performing targeted searches
+- **Observe** the results and iterate
+- **Answer** based on all gathered context
+
+This is particularly useful for questions that require:
+- Multiple pieces of information from different parts of your index
+- Iterative refinement of search queries
+- Complex reasoning that builds on previous findings
+
+## How It Works
+
+The ReAct agent follows a **Thought → Action → Observation** loop:
+
+1. **Thought**: The agent analyzes the question and determines what information is needed
+2. **Action**: The agent performs a search query based on its reasoning
+3. **Observation**: The agent reviews the search results
+4. **Iteration**: The process repeats until the agent has enough information or reaches the maximum iteration limit
+5. **Final Answer**: The agent synthesizes all gathered information into a comprehensive answer
+
+## Basic Usage
+
+### Command Line
+
+```bash
+# Basic usage
+leann react <index_name> "your question"
+
+# With custom LLM settings
+leann react my-index "What are the main features discussed?" \
+  --llm ollama \
+  --model qwen3:8b \
+  --max-iterations 5 \
+  --top-k 5
+```
+
+### Command Options
+
+- `index_name`: Name of the LEANN index to search
+- `query`: The question to research
+- `--llm`: LLM provider (`ollama`, `openai`, `anthropic`, `hf`, `simulated`) - default: `ollama`
+- `--model`: Model name (default: `qwen3:8b`)
+- `--host`: Override Ollama-compatible host (defaults to `LEANN_OLLAMA_HOST` or `OLLAMA_HOST`)
+- `--top-k`: Number of results per search iteration (default: `5`)
+- `--max-iterations`: Maximum number of search iterations (default: `5`)
+- `--api-base`: Base URL for OpenAI-compatible APIs
+- `--api-key`: API key for cloud LLM providers
+
+### Python API
+
+```python
+from leann import create_react_agent, LeannSearcher
+
+# Create a searcher
+searcher = LeannSearcher(index_path="path/to/index.leann")
+
+# Create the ReAct agent
+agent = create_react_agent(
+    index_path="path/to/index.leann",
+    llm_config={
+        "type": "ollama",
+        "model": "qwen3:8b",
+        "host": "http://localhost:11434"  # optional
+    },
+    max_iterations=5
+)
+
+# Run the agent
+answer = agent.run("What are the main topics covered in the documentation?", top_k=5)
+print(answer)
+
+# Access search history
+if agent.search_history:
+    print(f"\nSearch History ({len(agent.search_history)} iterations):")
+    for entry in agent.search_history:
+        print(f"  {entry['iteration']}. {entry['action']} ({entry['results_count']} results)")
+```
+
+## Example Use Cases
+
+### 1. Multi-faceted Questions
+
+```bash
+# Questions that need information from multiple sources
+leann react docs-index "What are the differences between HNSW and DiskANN backends, and when should I use each?"
+```
+
+The agent will:
+- First search for "HNSW backend features"
+- Then search for "DiskANN backend features"
+- Compare the results
+- Provide a comprehensive answer
+
+### 2. Iterative Research
+
+```bash
+# Questions requiring multiple search iterations
+leann react codebase-index "How does the embedding computation work and what optimizations are used?"
+```
+
+The agent will:
+- Search for "embedding computation"
+- Based on results, search for "embedding optimizations"
+- Refine queries based on findings
+- Synthesize the information
+
+### 3. Complex Reasoning
+
+```bash
+# Questions that require building understanding
+leann react research-index "What are the performance characteristics of different indexing strategies?"
+```
+
+## Comparison: `leann ask` vs `leann react`
+
+| Feature | `leann ask` | `leann react` |
+|---------|-------------|---------------|
+| **Search iterations** | Single search | Multiple iterations |
+| **Query refinement** | No | Yes, based on observations |
+| **Use case** | Simple Q&A | Complex, multi-faceted questions |
+| **Speed** | Faster | Slower (multiple searches) |
+| **Reasoning** | Direct answer | Iterative reasoning |
+
+### When to Use Each
+
+**Use `leann ask` when:**
+- You have a straightforward question
+- A single search should provide enough context
+- You want a quick answer
+
+**Use `leann react` when:**
+- Your question requires information from multiple sources
+- You need the agent to explore and refine its understanding
+- The answer requires synthesizing multiple pieces of information
+
+## Advanced Configuration
+
+### Custom LLM Providers
+
+```bash
+# Using OpenAI
+leann react my-index "question" \
+  --llm openai \
+  --model gpt-4 \
+  --api-base https://api.openai.com/v1 \
+  --api-key $OPENAI_API_KEY
+
+# Using Anthropic
+leann react my-index "question" \
+  --llm anthropic \
+  --model claude-3-opus-20240229 \
+  --api-key $ANTHROPIC_API_KEY
+```
+
+### Adjusting Search Parameters
+
+```bash
+# More results per iteration
+leann react my-index "question" --top-k 10
+
+# More iterations for complex questions
+leann react my-index "question" --max-iterations 10
+```
+
+## Understanding the Output
+
+When you run `leann react`, you'll see:
+
+1. **Question**: The original question being researched
+2. **Iteration logs**: Each search action and its results
+3. **Final Answer**: The synthesized answer based on all iterations
+4. **Search History**: Summary of all search iterations performed
+
+Example output:
+
+```
+🤖 Starting ReAct agent with index 'my-index'...
+Using qwen3:8b (ollama)
+
+🔍 Question: What are the main features of LEANN?
+
+🔍 Action: search("LEANN features")
+[Result 1] (Score: 0.923)
+LEANN is a vector database that saves 97% storage...
+
+🔍 Action: search("LEANN storage optimization")
+[Result 1] (Score: 0.891)
+LEANN uses compact storage and recomputation...
+
+✅ Final Answer:
+LEANN is a vector database with several key features:
+1. 97% storage savings compared to traditional vector databases
+2. Compact storage with recomputation capabilities
+3. Support for multiple backends (HNSW and DiskANN)
+...
+
+📊 Search History (2 iterations):
+  1. search("LEANN features") (5 results)
+  2. search("LEANN storage optimization") (5 results)
+```
+
+## Tips for Best Results
+
+1. **Be specific**: Clear, specific questions work better than vague ones
+2. **Adjust iterations**: Complex questions may need more iterations (increase `--max-iterations`)
+3. **Monitor history**: Check the search history to understand the agent's reasoning
+4. **Use appropriate models**: Larger models generally provide better reasoning, but are slower
+5. **Index quality**: Ensure your index is well-built with relevant content
+
+## Limitations
+
+- **Speed**: Multiple iterations make ReAct slower than single-search queries
+- **Cost**: More LLM calls mean higher costs for cloud providers
+- **Complexity**: Very complex questions may still require human review
+- **Model dependency**: Reasoning quality depends on the LLM's capabilities
+
+## Future Enhancements
+
+This is the first implementation (1/N) of Deep-Research integration. Future enhancements may include:
+- Web search integration for external information
+- More sophisticated reasoning strategies
+- Parallel search execution
+- Better query optimization
+
+## Related Documentation
+
+- [Basic Usage Guide](../README.md)
+- [CLI Reference](configuration-guide.md)
+- [Embedding Models](normalized_embeddings.md)

docs/roadmap.md

@@ -0,0 +1,67 @@
+# LEANN Roadmap
+
+LEANN aims to be a **personal knowledge layer** — not just a storage-efficient vector database, but a unified, always-up-to-date knowledge base that runs entirely on your own machine. It connects your code, images, and personal data (documents, emails, browser history, chats) into a single multimodal search interface.
+
+Contributions and feedback are welcome. Join our [Slack](https://join.slack.com/t/leann-e2u9779/shared_invite/zt-3ol2ww9ic-Eg_kB8omwe6xmYVd0epr4Q) to discuss.
+
+---
+
+## Completed
+
+- [x] HNSW backend integration
+- [x] DiskANN backend with MIPS/L2/Cosine support
+- [x] Real-time embedding pipeline
+- [x] Memory-efficient graph pruning
+- [x] IVF backend with incremental add/remove (#231, #89, #141)
+- [x] Merkle tree file-change detection — `leann watch` (#41)
+
+---
+
+## P0 — Core (Q1 2026)
+
+### LEANN MCP — The Best Code Retrieval MCP
+
+The primary near-term goal: make LEANN the go-to MCP server for code-aware AI assistants. This means **dynamic updates** (your index stays current as you edit code), **rich code context** (AST-aware chunking that understands functions, classes, and modules — not just raw text), and a **dead-simple interface** (one command to build, automatic incremental updates, zero configuration for common setups).
+
+- [x] IVF backend — incremental add/remove without full rebuild (#231, #89, #141)
+- [x] Merkle tree file-change detection — `leann watch` for automatic re-indexing on file changes (#41)
+- [ ] Cold start optimization — faster first-build experience for new users (#166, #177)
+- [ ] Live index updates — push index changes as files are saved, not just on rebuild
+- [ ] Smarter code context — cross-file symbol resolution, call graph awareness, import tracking
+
+### Search Quality
+
+- [ ] Hybrid search — combine dense vector retrieval with sparse keyword matching (BM25) for better recall on exact identifiers and variable names (#233, #90)
+
+### Documentation
+
+- [ ] ReadTheDocs — hosted documentation site (#234)
+- [ ] Benchmarks — recall@k, latency, and storage comparisons across backends
+
+---
+
+## P1 (Q2 2026)
+
+### Multimodal
+
+- [ ] Video retrieval (#160)
+- [ ] CLIP support — image-text cross-modal search (#94)
+- [ ] OCR — extract text from images/scanned documents (#158)
+
+### Platform & Distribution
+
+- [ ] Windows support (#14)
+- [ ] Web UI (#229)
+
+### Applications & Integrations
+
+- [ ] Agent + Deep research (#104)
+- [ ] Local Cursor — local model + local retrieval for code assistance (#47)
+- [ ] LlamaIndex integration (#217)
+- [ ] Obsidian support (#96)
+
+---
+
+## Contributing
+
+If you're interested in working on any of the items above, please reach out to [@yichuan-w](https://github.com/yichuan-w) or [@andylizf](https://github.com/andylizf). See [CONTRIBUTING.md](CONTRIBUTING.md) for the full contributor workflow.

docs/slack-setup-guide.md

@@ -0,0 +1,395 @@
+# Slack Integration Setup Guide
+
+This guide provides step-by-step instructions for setting up Slack integration with LEANN.
+
+## Overview
+
+LEANN's Slack integration uses MCP (Model Context Protocol) servers to fetch and index your Slack messages for RAG (Retrieval-Augmented Generation). This allows you to search through your Slack conversations using natural language queries.
+
+## Prerequisites
+
+1. **Slack Workspace Access**: You need admin or owner permissions in your Slack workspace to create apps and configure OAuth tokens.
+
+2. **Slack MCP Server**: Install a Slack MCP server (e.g., `slack-mcp-server` via npm)
+
+3. **LEANN**: Ensure you have LEANN installed and working
+
+## Step 1: Create a Slack App
+
+### 1.1 Go to Slack API Dashboard
+
+1. Visit [https://api.slack.com/apps](https://api.slack.com/apps)
+2. Click **"Create New App"**
+3. Choose **"From scratch"**
+4. Enter your app name (e.g., "LEANN Slack Integration")
+5. Select your workspace
+6. Click **"Create App"**
+
+### 1.2 Configure App Permissions
+
+#### Token Scopes
+
+1. In your app dashboard, go to **"OAuth & Permissions"** in the left sidebar
+2. Scroll down to **"Scopes"** section
+3. Under **"Bot Token Scopes & OAuth Scope"**, click **"Add an OAuth Scope"**
+4. Add the following scopes:
+   - `channels:read` - Read public channel information
+   - `channels:history` - Read messages in public channels
+   - `groups:read` - Read private channel information
+   - `groups:history` - Read messages in private channels
+   - `im:read` - Read direct message information
+   - `im:history` - Read direct messages
+   - `mpim:read` - Read group direct message information
+   - `mpim:history` - Read group direct messages
+   - `users:read` - Read user information
+   - `team:read` - Read workspace information
+
+#### App-Level Tokens (Optional)
+
+Some MCP servers may require app-level tokens:
+
+1. Go to **"Basic Information"** in the left sidebar
+2. Scroll down to **"App-Level Tokens"**
+3. Click **"Generate Token and Scopes"**
+4. Enter a name (e.g., "LEANN Integration")
+5. Add the `connections:write` scope
+6. Click **"Generate"**
+7. Copy the token (starts with `xapp-`)
+
+### 1.3 Install App to Workspace
+
+1. Go to **"OAuth & Permissions"** in the left sidebar
+2. Click **"Install to Workspace"**
+3. Review the permissions and click **"Allow"**
+4. Copy the **"Bot User OAuth Token"** (starts with `xoxb-`)
+5. Copy the **"User OAuth Token"** (starts with `xoxp-`)
+
+## Step 2: Install Slack MCP Server
+
+### Option A: Using npm (Recommended)
+
+```bash
+# Install globally
+npm install -g slack-mcp-server
+
+# Or install locally
+npm install slack-mcp-server
+```
+
+### Option B: Using npx (No installation required)
+
+```bash
+# Use directly without installation
+npx slack-mcp-server
+```
+
+## Step 3: Install and Configure Ollama (for Real LLM Responses)
+
+### 3.1 Install Ollama
+
+```bash
+# Install Ollama using Homebrew (macOS)
+brew install ollama
+
+# Or download from https://ollama.ai/
+```
+
+### 3.2 Start Ollama Service
+
+```bash
+# Start Ollama as a service
+brew services start ollama
+
+# Or start manually
+ollama serve
+```
+
+### 3.3 Pull a Model
+
+```bash
+# Pull a lightweight model for testing
+ollama pull llama3.2:1b
+
+# Verify the model is available
+ollama list
+```
+
+## Step 4: Configure Environment Variables
+
+Create a `.env` file or set environment variables:
+
+```bash
+# Required: User OAuth Token
+SLACK_OAUTH_TOKEN=xoxp-your-user-oauth-token-here
+
+# Optional: App-Level Token (if your MCP server requires it)
+SLACK_APP_TOKEN=xapp-your-app-token-here
+
+# Optional: Workspace-specific settings
+SLACK_WORKSPACE_ID=T1234567890  # Your workspace ID (optional)
+```
+
+## Step 5: Test the Setup
+
+### 5.1 Test MCP Server Connection
+
+```bash
+python -m apps.slack_rag \
+  --mcp-server "slack-mcp-server" \
+  --test-connection \
+  --workspace-name "Your Workspace Name"
+```
+
+This will test the connection and list available tools without indexing any data.
+
+### 5.2 Index a Specific Channel
+
+```bash
+python -m apps.slack_rag \
+  --mcp-server "slack-mcp-server" \
+  --workspace-name "Your Workspace Name" \
+  --channels general \
+  --query "What did we discuss about the project?"
+```
+
+### 5.3 Real RAG Query Examples
+
+This section demonstrates successful Slack RAG integration queries against the Sky Lab Computing workspace's "random" channel. The system successfully retrieves actual conversation messages and performs semantic search with high relevance scores, including finding specific research paper announcements and technical discussions.
+
+### Example 1: Advisor Models Query
+
+**Query:** "train black-box models to adopt to your personal data"
+
+This query demonstrates the system's ability to find specific research announcements about training black-box models for personal data adaptation.
+
+![Advisor Models Query - Command Setup](videos/slack_integration_1.1.png)
+
+![Advisor Models Query - Search Results](videos/slack_integration_1.2.png)
+
+![Advisor Models Query - LLM Response](videos/slack_integration_1.3.png)
+
+### Example 2: Barbarians at the Gate Query
+
+**Query:** "AI-driven research systems ADRS"
+
+This query demonstrates the system's ability to find specific research announcements about AI-driven research systems and algorithm discovery.
+
+![Barbarians Query - Command Setup](videos/slack_integration_2.1.png)
+
+![Barbarians Query - Search Results](videos/slack_integration_2.2.png)
+
+![Barbarians Query - LLM Response](videos/slack_integration_2.3.png)
+
+### Prerequisites
+
+- Bot is installed in the Sky Lab Computing workspace and invited to the target channel (run `/invite @YourBotName` in the channel if needed)
+- Bot token available and exported in the same terminal session
+
+### Commands
+
+1) Set the workspace token for this shell
+
+```bash
+export SLACK_MCP_XOXP_TOKEN="xoxp-***-redacted-***"
+```
+
+2) Run queries against the "random" channel by channel ID (C0GN5BX0F)
+
+**Advisor Models Query:**
+```bash
+python -m apps.slack_rag \
+  --mcp-server "slack-mcp-server" \
+  --workspace-name "Sky Lab Computing" \
+  --channels C0GN5BX0F \
+  --max-messages-per-channel 100000 \
+  --query "train black-box models to adopt to your personal data" \
+  --llm ollama \
+  --llm-model "llama3.2:1b" \
+  --llm-host "http://localhost:11434" \
+  --no-concatenate-conversations
+```
+
+**Barbarians at the Gate Query:**
+```bash
+python -m apps.slack_rag \
+  --mcp-server "slack-mcp-server" \
+  --workspace-name "Sky Lab Computing" \
+  --channels C0GN5BX0F \
+  --max-messages-per-channel 100000 \
+  --query "AI-driven research systems ADRS" \
+  --llm ollama \
+  --llm-model "llama3.2:1b" \
+  --llm-host "http://localhost:11434" \
+  --no-concatenate-conversations
+```
+
+These examples demonstrate the system's ability to find and retrieve specific research announcements and technical discussions from the conversation history, showcasing the power of semantic search in Slack data.
+
+3) Optional: Ask a broader question
+
+```bash
+python test_channel_by_id_or_name.py \
+  --channel-id C0GN5BX0F \
+  --workspace-name "Sky Lab Computing" \
+  --query "What is LEANN about?"
+```
+
+Notes:
+- If you see `not_in_channel`, invite the bot to the channel and re-run.
+- If you see `channel_not_found`, confirm the channel ID and workspace.
+- Deep search via server-side “search” tools may require additional Slack scopes; the example above performs client-side filtering over retrieved history.
+
+## Common Issues and Solutions
+
+### Issue 1: "users cache is not ready yet" Error
+
+**Problem**: You see this warning:
+```
+WARNING - Failed to fetch messages from channel random: Failed to fetch messages: {'code': -32603, 'message': 'users cache is not ready yet, sync process is still running... please wait'}
+```
+
+**Solution**: This is a common timing issue. The LEANN integration now includes automatic retry logic:
+
+1. **Wait and Retry**: The system will automatically retry with exponential backoff (2s, 4s, 8s, etc.)
+2. **Increase Retry Parameters**: If needed, you can customize retry behavior:
+   ```bash
+   python -m apps.slack_rag \
+     --mcp-server "slack-mcp-server" \
+     --max-retries 10 \
+     --retry-delay 3.0 \
+     --channels general \
+     --query "Your query here"
+   ```
+3. **Keep MCP Server Running**: Start the MCP server separately and keep it running:
+   ```bash
+   # Terminal 1: Start MCP server
+   slack-mcp-server
+
+   # Terminal 2: Run LEANN (it will connect to the running server)
+   python -m apps.slack_rag --mcp-server "slack-mcp-server" --channels general --query "test"
+   ```
+
+### Issue 2: "No message fetching tool found"
+
+**Problem**: The MCP server doesn't have the expected tools.
+
+**Solution**:
+1. Check if your MCP server is properly installed and configured
+2. Verify your Slack tokens are correct
+3. Try a different MCP server implementation
+4. Check the MCP server documentation for required configuration
+
+### Issue 3: Permission Denied Errors
+
+**Problem**: You get permission errors when trying to access channels.
+
+**Solutions**:
+1. **Check Bot Permissions**: Ensure your bot has been added to the channels you want to access
+2. **Verify Token Scopes**: Make sure you have all required scopes configured
+3. **Channel Access**: For private channels, the bot needs to be explicitly invited
+4. **Workspace Permissions**: Ensure your Slack app has the necessary workspace permissions
+
+### Issue 4: Empty Results
+
+**Problem**: No messages are returned even though the channel has messages.
+
+**Solutions**:
+1. **Check Channel Names**: Ensure channel names are correct (without the # symbol)
+2. **Verify Bot Access**: Make sure the bot can access the channels
+3. **Check Date Ranges**: Some MCP servers have limitations on message history
+4. **Increase Message Limits**: Try increasing the message limit:
+   ```bash
+   python -m apps.slack_rag \
+     --mcp-server "slack-mcp-server" \
+     --channels general \
+     --max-messages-per-channel 1000 \
+     --query "test"
+   ```
+
+## Advanced Configuration
+
+### Custom MCP Server Commands
+
+If you need to pass additional parameters to your MCP server:
+
+```bash
+python -m apps.slack_rag \
+  --mcp-server "slack-mcp-server --token-file /path/to/tokens.json" \
+  --workspace-name "Your Workspace" \
+  --channels general \
+  --query "Your query"
+```
+
+### Multiple Workspaces
+
+To work with multiple Slack workspaces, you can:
+
+1. Create separate apps for each workspace
+2. Use different environment variables
+3. Run separate instances with different configurations
+
+### Performance Optimization
+
+For better performance with large workspaces:
+
+```bash
+python -m apps.slack_rag \
+  --mcp-server "slack-mcp-server" \
+  --workspace-name "Your Workspace" \
+  --max-messages-per-channel 500 \
+  --no-concatenate-conversations \
+  --query "Your query"
+```
+---
+
+## Troubleshooting Checklist
+
+- [ ] Slack app created with proper permissions
+- [ ] Bot token (xoxb-) copied correctly
+- [ ] App-level token (xapp-) created if needed
+- [ ] MCP server installed and accessible
+- [ ] Ollama installed and running (`brew services start ollama`)
+- [ ] Ollama model pulled (`ollama pull llama3.2:1b`)
+- [ ] Environment variables set correctly
+- [ ] Bot invited to relevant channels
+- [ ] Channel names specified without # symbol
+- [ ] Sufficient retry attempts configured
+- [ ] Network connectivity to Slack APIs
+
+## Getting Help
+
+If you continue to have issues:
+
+1. **Check Logs**: Look for detailed error messages in the console output
+2. **Test MCP Server**: Use `--test-connection` to verify the MCP server is working
+3. **Verify Tokens**: Double-check that your Slack tokens are valid and have the right scopes
+4. **Check Ollama**: Ensure Ollama is running (`ollama serve`) and the model is available (`ollama list`)
+5. **Community Support**: Reach out to the LEANN community for help
+
+## Example Commands
+
+### Basic Usage
+```bash
+# Test connection
+python -m apps.slack_rag --mcp-server "slack-mcp-server" --test-connection
+
+# Index specific channels
+python -m apps.slack_rag \
+  --mcp-server "slack-mcp-server" \
+  --workspace-name "My Company" \
+  --channels general random \
+  --query "What did we decide about the project timeline?"
+```
+
+### Advanced Usage
+```bash
+# With custom retry settings
+python -m apps.slack_rag \
+  --mcp-server "slack-mcp-server" \
+  --workspace-name "My Company" \
+  --channels general \
+  --max-retries 10 \
+  --retry-delay 5.0 \
+  --max-messages-per-channel 2000 \
+  --query "Show me all decisions made in the last month"
+```

docs/ultimate_goal.md

@@ -0,0 +1,41 @@
+# LEANN — Long-Term Vision
+
+## The Best Personal Data Management Platform
+
+LEANN's ultimate goal is to be the **unified personal knowledge layer** that lives on your machine. Not a cloud service. Not a SaaS product. A local-first system that understands everything you've ever worked on — code, documents, emails, chats, browser history, images — and makes it all instantly searchable and usable.
+
+### 1. Continuous Learning from Your Context
+
+LEANN should get smarter over time by continuously ingesting and indexing your data as you produce it:
+
+- **Always-on indexing**: Watch your filesystem, email, browser, and chat sources. Incrementally update indexes as new data arrives — no manual rebuilds.
+- **Cross-source connections**: Surface relationships across data sources. A Slack conversation about a bug should link to the relevant code change, the related email thread, and the document that describes the feature.
+- **Temporal awareness**: Understand when things happened. "What was I working on last Tuesday?" should be a searchable query.
+- **Personalized ranking**: Learn from your search patterns and usage to rank results by relevance to *you*, not just semantic similarity.
+
+### 2. The Best MCP for Code Retrieval
+
+AI coding assistants are only as good as the context they receive. LEANN aims to be the best context provider:
+
+- **AST-aware chunking**: Understand code structure — functions, classes, modules — not just raw text blocks. Retrieve semantically meaningful units.
+- **Dynamic index updates**: The index stays current as you edit. No stale results. No manual rebuilds. Save a file, and the index reflects the change within seconds.
+- **Cross-file understanding**: Resolve symbols across files. When you search for a function, also surface its callers, its tests, and the types it depends on.
+- **Dead-simple interface**: One command to build (`leann build`), automatic incremental updates, MCP server that any AI assistant can plug into with zero configuration.
+- **Repository-scale search**: Handle monorepos and large codebases without breaking a sweat. The storage efficiency (97% reduction vs. traditional vector DBs) makes this feasible on a laptop.
+
+### 3. Multimodal Knowledge Base
+
+Text is just the beginning:
+
+- **Image search**: CLIP-based retrieval for screenshots, diagrams, photos. "Find the architecture diagram from last month."
+- **Video retrieval**: Index video content — lectures, meetings, screen recordings — and search by what was said or shown.
+- **OCR pipeline**: Extract and index text from scanned documents, handwritten notes, whiteboard photos.
+- **Unified search**: One query searches across all modalities. The answer might be in a PDF, a screenshot, a code comment, or a Slack message.
+
+### 4. Platform for Personal AI
+
+Looking further ahead, LEANN becomes the memory and retrieval layer for personal AI agents:
+
+- **Agent memory**: AI agents that remember your preferences, past decisions, and project context across sessions.
+- **Deep research**: Agents that can search your entire personal knowledge base to answer complex questions, synthesize information, and generate insights.
+- **Proactive suggestions**: Surface relevant context before you ask for it — "You discussed this exact problem with a colleague 3 months ago, here's what you decided."

docs/user-scripts.md

@@ -0,0 +1,95 @@
+# Data Source Indexing Commands
+
+LEANN provides `index-*` CLI commands for indexing common personal data sources. Each command reads from a specific data source and builds a searchable LEANN index.
+
+## Available Commands
+
+| Command | Description | Platform |
+|---------|-------------|----------|
+| `leann index-browser [chrome\|brave]` | Browser history | macOS |
+| `leann index-email` | Apple Mail | macOS |
+| `leann index-calendar` | Apple Calendar | macOS |
+| `leann index-imessage` | iMessage conversations | macOS |
+| `leann index-wechat --export-dir <path>` | WeChat chat history | Any |
+| `leann index-chatgpt --export-path <path>` | ChatGPT export | Any |
+| `leann index-claude --export-path <path>` | Claude export | Any |
+
+## Common Options
+
+All `index-*` commands accept these options:
+
+```bash
+--index-name NAME          # Custom index name (each command has a sensible default)
+--embedding-model MODEL    # Embedding model (default: facebook/contriever)
+--embedding-mode MODE      # Backend: sentence-transformers, openai, mlx, ollama
+--max-count N              # Max items to index (default: 1000)
+--no-recompute             # Store full embeddings instead of using recomputation
+```
+
+## Examples
+
+### Index Chrome browser history
+
+```bash
+leann index-browser chrome
+leann index-browser brave --index-name brave_history
+```
+
+### Index Apple Mail
+
+```bash
+leann index-email
+```
+
+### Index iMessage
+
+```bash
+leann index-imessage
+```
+
+### Index Apple Calendar
+
+```bash
+leann index-calendar
+```
+
+### Index ChatGPT or Claude exports
+
+```bash
+# ChatGPT: export from https://chat.openai.com → Settings → Export data
+leann index-chatgpt --export-path ~/Downloads/chatgpt-export.zip
+
+# Claude: export from https://claude.ai → Settings → Export data
+leann index-claude --export-path ~/Downloads/claude-export.json
+```
+
+### Index WeChat
+
+```bash
+# Requires exported JSON files from wechat-exporter
+leann index-wechat --export-dir ~/wechat-export/
+```
+
+## Daily Automation
+
+You can schedule indexing with cron for automatic daily updates:
+
+```bash
+# Edit crontab
+crontab -e
+
+# Add entries (runs at 2 AM daily):
+0 2 * * * cd /path/to/LEANN && leann index-browser chrome
+5 2 * * * cd /path/to/LEANN && leann index-email
+10 2 * * * cd /path/to/LEANN && leann index-imessage
+```
+
+## Searching Indexed Data
+
+After indexing, search with the standard `leann search` command:
+
+```bash
+leann search browser_history "github pull request review"
+leann search email "meeting notes from last week"
+leann search imessage "dinner plans"
+```