chore: import upstream snapshot with attribution

docs/content/Guides/Architecture.mdx

@@ -0,0 +1,171 @@
+---
+title: Architecture
+description: High-level architecture of DocsGPT
+---
+
+## Introduction
+
+DocsGPT is designed as a modular and scalable application for knowledge based GenAI system. This document outlines the high-level architecture of DocsGPT, highlighting its key components.
+
+## High-Level Architecture
+
+This diagram provides a bird's-eye view of the DocsGPT architecture, illustrating the main components and their interactions.
+
+```mermaid
+flowchart LR
+    User["User"] --> Frontend["Frontend (React/Vite)"]
+    Frontend --> Backend["Backend API (Flask)"]
+    Backend --> LLM["LLM Integration Layer"] & VectorStore["Vector Stores"] & TaskQueue["Task Queue (Celery)"] & Databases["Databases (Postgres, Redis)"]
+    LLM -- Cloud APIs / Local Engines --> InferenceEngine["Inference Engine"]
+    VectorStore -- Document Embeddings --> Indexes[("Indexes")]
+    TaskQueue -- Asynchronous Tasks --> DocumentIngestion["Document Ingestion"]
+
+    style Frontend fill:#AA00FF,color:#FFFFFF
+    style Backend fill:#AA00FF,color:#FFFFFF
+    style LLM fill:#AA00FF,color:#FFFFFF
+    style TaskQueue fill:#AA00FF,color:#FFFFFF,stroke:#AA00FF
+    style DocumentIngestion fill:#AA00FF,color:#FFFFFF,stroke:none
+```
+
+## Component Descriptions
+
+### 1. Frontend (React/Vite)
+
+*   **Technology:** Built using React and Vite.
+*   **Responsibility:** This is the user interface of DocsGPT, providing users with an UI to ask questions and receive answers, configure prompts, tools and other settings. It handles user input, displays conversation history, shows sources, and manages settings.
+*   **Key Features:**
+    *   Clean and responsive UI.
+    *   Simple static client-side rendering.
+    *   Manages conversation state and settings.
+    *   Communicates with the Backend API for data retrieval and processing.
+
+### 2. Backend API (Flask)
+
+*   **Technology:** Implemented using Flask (Python).
+*   **Responsibility:** The Backend API serves as the core logic and orchestration layer of DocsGPT. It receives requests from the Frontend, Extensions or API clients, processes them, and coordinates interactions between different components.
+*   **Key Features:**
+    *   API endpoints for handling user queries, document uploads, and settings configurations.
+    *   Manages the overall application flow and logic.
+    *   Integrates with the LLM Integration Layer, Vector Stores, Task Queue, Tools, Agents and Databases.
+    *   Provides Swagger documentation for API endpoints.
+
+### 3. LLM Integration Layer (Part of backend)
+
+*   **Technology:** Supports multiple LLM APIs and local engines.
+*   **Responsibility:** This layer provides an abstraction for interacting with Large Language Models (LLMs).
+*   **Key Features:**
+    *   Supports LLMs from OpenAI, Google, Anthropic, Groq, HuggingFace Inference API, Azure OpenAI, also compatible with local models like Ollama, LLaMa.cpp, Text Generation Inference (TGI), SGLang, vLLM, Aphrodite, FriendliAI, and LMDeploy.
+    *   Manages API key handling and request formatting and Tool formatting.
+    *   Offers caching mechanisms to improve response times and reduce API usage.
+    *   Handles streaming responses for a more interactive user experience.
+
+### 4. Vector Stores (Part of backend)
+
+*   **Technology:** Supports multiple vector databases.
+*   **Responsibility:** Vector Stores are used to store and retrieve vector embeddings of document chunks. This enables semantic search and retrieval of relevant document snippets in response to user queries.
+*   **Key Features:**
+    *   Supports vector databases including FAISS, Elasticsearch, Qdrant, Milvus, MongoDB Atlas Vector Search, and pgvector.
+    *   Provides storage and indexing of high-dimensional vector embeddings.
+    *   Enables editing and updating of vector indexes including specific chunks.
+
+### 4a. Retrieval Layer (Part of backend)
+
+*   **Responsibility:** Sits between the API and the vector stores and decides *how* context is fetched for each question. Rather than a single similarity search, a **retriever dispatcher** groups the question's sources by their configured retriever and runs each under a shared token budget, then applies post-retrieval stages before answering.
+*   **Key Features:**
+    *   Multiple retrievers: `classic` (vector similarity), `hybrid` (vector + full-text keyword fusion), and `graphrag` (knowledge-graph / Personalized PageRank).
+    *   Optional query rephrasing before retrieval.
+    *   Optional post-retrieval stages such as LLM pre-screening (map-reduce relevance filtering).
+    *   Per-source behavior via the [per-source configuration](/Sources/Per-source-configuration) contract, so different sources can be chunked and retrieved differently.
+    *   See [GraphRAG](/Sources/GraphRAG) for the knowledge-graph retrieval path.
+
+### 5. Parser Integration Layer (Part of backend)
+
+*   **Technology:** Supports multiple formats for file processing and remote source uploading.
+*   **Responsibility:** Parser Integration Layer handles uploading, parsing, chunking, embedding, and indexing documents.
+*   **Key Features:**
+    *   Supports various document formats (PDF, DOCX, TXT, etc.) and remote sources (web URLs, sitemaps).
+    *   Handles document parsing, text chunking, and embedding generation.
+    *   Utilizes Celery for asynchronous processing, ensuring efficient handling of large documents.
+
+### 6. Task Queue (Celery)
+
+*   **Technology:** Celery with Redis as broker and backend.
+*   **Responsibility:** Celery handles asynchronous task processing, for long-running operations such as document ingestion and indexing. This ensures that the main application remains responsive and efficient.
+*   **Key Features:**
+    *   Manages background tasks for document processing and indexing.
+    *   Improves application responsiveness by offloading heavy tasks.
+    *   Enhances scalability and reliability through distributed task processing.
+
+### 7. Databases (Postgres, Redis)
+
+*   **Technology:** PostgreSQL and Redis.
+*   **Responsibility:** Databases are used for persistent data storage and caching. PostgreSQL stores structured user data such as conversations, agents, prompts, sources, attachments, workflows, logs, token usage, user settings, and API keys. Redis is used as a cache and as the message broker/result backend for Celery.
+*   **Note:** MongoDB is no longer used for user data. It remains an **optional** backend for the vector store (`VECTOR_STORE=mongodb`, i.e. Mongo Atlas Vector Search) and as the source database for the one-shot `scripts/db/backfill.py` migration from legacy installs.
+
+## Request Flow Diagram
+
+This diagram illustrates the sequence of steps involved when a user submits a question to DocsGPT.
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant Frontend
+    participant BackendAPI
+    participant RetrievalLayer
+    participant LLMIntegrationLayer
+    participant VectorStores
+    participant InferenceEngine
+
+    User->>Frontend: User asks a question
+    Frontend->>BackendAPI: API Request (Question)
+    BackendAPI->>RetrievalLayer: Dispatch retrieval (per-source: classic / hybrid / graphrag)
+    RetrievalLayer->>VectorStores: Fetch candidates (vector / keyword / graph)
+    VectorStores-->>RetrievalLayer: Return candidates
+    RetrievalLayer-->>BackendAPI: Ranked, optionally pre-screened chunks
+    BackendAPI->>LLMIntegrationLayer: Send question and document chunks
+    LLMIntegrationLayer->>InferenceEngine: LLM API Request (Prompt + Context)
+    InferenceEngine-->>LLMIntegrationLayer: LLM API Response (Answer)
+    LLMIntegrationLayer-->>BackendAPI: Return Answer
+    BackendAPI->>Frontend: API Response (Answer)
+    Frontend->>User: Display Answer
+
+    Note over Frontend,BackendAPI: Data flow is simplified for clarity
+```
+
+## Deployment Architecture
+
+DocsGPT is designed to be deployed using Docker and Kubernetes, here is a quick overview of a simple k8s deployment.
+
+```mermaid
+graph LR
+    subgraph Kubernetes Cluster
+        subgraph Nodes
+            subgraph Node 1
+                FrontendPod[Frontend Pod]
+                BackendAPIPod[Backend API Pod]
+            end
+            subgraph Node 2
+                CeleryWorkerPod[Celery Worker Pod]
+                RedisPod[Redis Pod]
+            end
+            subgraph Node 3
+                PostgresPod[Postgres Pod]
+                VectorStorePod[Vector Store Pod]
+            end
+        end
+        LoadBalancer[Load Balancer] --> docsgpt-frontend-service[docsgpt-frontend-service]
+        LoadBalancer --> docsgpt-api-service[docsgpt-api-service]
+        docsgpt-frontend-service --> FrontendPod
+        docsgpt-api-service --> BackendAPIPod
+        BackendAPIPod --> CeleryWorkerPod
+        BackendAPIPod --> RedisPod
+        BackendAPIPod --> PostgresPod
+        BackendAPIPod --> VectorStorePod
+        CeleryWorkerPod --> RedisPod
+        BackendAPIPod --> InferenceEngine[(Inference Engine)]
+        VectorStorePod --> Indexes[(Indexes)]
+        PostgresPod --> Data[(Data)]
+        RedisPod --> Cache[(Cache)]
+    end
+    User[User] --> LoadBalancer
+```

docs/content/Guides/Customising-prompts.mdx

@@ -0,0 +1,453 @@
+---
+title: Customizing Prompts
+description: This guide explains how to customize prompts in DocsGPT using the new template-based system with dynamic variable injection.
+---
+
+import Image from 'next/image'
+
+# Customizing Prompts in DocsGPT
+
+Customizing prompts for DocsGPT gives you powerful control over the AI's behavior and responses. With the new template-based system, you can inject dynamic context through organized namespaces, making prompts flexible and maintainable without hardcoding values.
+
+## Quick Start
+
+1. Navigate to `SideBar -> Settings`.
+2. In Settings, select the `Active Prompt` to see various prompt styles.
+3. Click on the `edit icon` on your chosen prompt to customize it.
+
+### Video Demo
+<Image src="/prompts.gif" alt="prompts" width={800} height={500} />
+
+---
+
+## Template-Based Prompt System
+
+DocsGPT now uses **Jinja2 templating** with four organized namespaces for dynamic variable injection:
+
+### Available Namespaces
+
+#### 1. **`system`** - System Metadata
+Access system-level information:
+
+```jinja
+{{ system.date }}         # Current date (YYYY-MM-DD)
+{{ system.time }}         # Current time (HH:MM:SS)
+{{ system.timestamp }}    # ISO 8601 timestamp
+{{ system.request_id }}   # Unique request identifier
+{{ system.user_id }}      # Current user ID
+```
+
+#### 2. **`source`** - Retrieved Documents
+Access RAG (Retrieval-Augmented Generation) document context:
+
+```jinja
+{{ source.content }}      # Concatenated document content
+{{ source.summaries }}    # Alias for content (backward compatible)
+{{ source.documents }}    # List of document objects
+{{ source.count }}        # Number of retrieved documents
+```
+
+#### 3. **`passthrough`** - Request Parameters
+Access custom parameters passed in the API request:
+
+```jinja
+{{ passthrough.company }}     # Custom field from request
+{{ passthrough.user_name }}   # User-provided data
+{{ passthrough.context }}     # Any custom parameter
+```
+
+To use passthrough data, send it in your API request:
+```json
+{
+  "question": "What is the pricing?",
+  "passthrough": {
+    "company": "Acme Corp",
+    "user_name": "Alice",
+    "plan_type": "enterprise"
+  }
+}
+```
+
+#### 4. **`tools`** - Pre-fetched Tool Data
+Access results from tools that run before the agent (like memory tool):
+
+```jinja
+{{ tools.memory.root }}       # Memory tool directory listing
+{{ tools.memory.available }}  # Boolean: is memory available
+```
+
+---
+
+## Example Prompts
+
+### Basic Prompt with Documents
+```jinja
+You are a helpful AI assistant for DocsGPT.
+
+Current date: {{ system.date }}
+
+Use the following documents to answer the question:
+
+{{ source.content }}
+
+Provide accurate, helpful answers with code examples when relevant.
+```
+
+### Advanced Prompt with All Namespaces
+```jinja
+You are an AI assistant for {{ passthrough.company }}.
+
+**System Info:**
+- Date: {{ system.date }}
+- Request ID: {{ system.request_id }}
+
+**User Context:**
+- User: {{ passthrough.user_name }}
+- Role: {{ passthrough.role }}
+
+**Available Documents ({{ source.count }}):**
+{{ source.content }}
+
+**Memory Context:**
+{% if tools.memory.available %}
+{{ tools.memory.root }}
+{% else %}
+No saved context available.
+{% endif %}
+
+Please provide detailed, accurate answers based on the documents above.
+```
+
+### Conditional Logic Example
+```jinja
+You are a DocsGPT assistant.
+
+{% if source.count > 0 %}
+I found {{ source.count }} relevant document(s):
+
+{{ source.content }}
+
+Base your answer on these documents.
+{% else %}
+No documents were found. Please answer based on your general knowledge.
+{% endif %}
+```
+
+---
+
+## Migration Guide
+
+### Legacy Format (Still Supported)
+The old `{summaries}` format continues to work for backward compatibility:
+
+```markdown
+You are a helpful assistant.
+
+Documents:
+{summaries}
+```
+
+This will automatically substitute `{summaries}` with document content.
+
+### New Template Format (Recommended)
+Migrate to the new template syntax for more flexibility:
+
+```jinja
+You are a helpful assistant.
+
+Documents:
+{{ source.content }}
+```
+
+**Migration mapping:**
+- `{summaries}` → `{{ source.content }}` or `{{ source.summaries }}`
+
+---
+
+## Best Practices
+
+### 1. **Use Descriptive Context**
+```jinja
+**Retrieved Documents:**
+{{ source.content }}
+
+**User Query Context:**
+- Company: {{ passthrough.company }}
+- Department: {{ passthrough.department }}
+```
+
+### 2. **Handle Missing Data Gracefully**
+```jinja
+{% if passthrough.user_name %}
+Hello {{ passthrough.user_name }}!
+{% endif %}
+```
+
+### 3. **Leverage Memory for Continuity**
+```jinja
+{% if tools.memory.available %}
+**Previous Context:**
+{{ tools.memory.root }}
+{% endif %}
+
+**Current Question:**
+Please consider the above context when answering.
+```
+
+### 4. **Add Clear Instructions**
+```jinja
+You are a technical support assistant.
+
+**Guidelines:**
+1. Always reference the documents below
+2. Provide step-by-step instructions
+3. Include code examples when relevant
+
+**Reference Documents:**
+{{ source.content }}
+```
+
+---
+
+## Advanced Features
+
+### Looping Over Documents
+```jinja
+{% for doc in source.documents %}
+**Source {{ loop.index }}:** {{ doc.filename }}
+{{ doc.text }}
+
+{% endfor %}
+```
+
+### Date-Based Behavior
+```jinja
+{% if system.date > "2025-01-01" %}
+Note: This is information from 2025 or later.
+{% endif %}
+```
+
+### Custom Formatting
+```jinja
+**Request Information**
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+• Request ID: {{ system.request_id }}
+• User: {{ passthrough.user_name | default("Guest") }}
+• Time: {{ system.time }}
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+```
+
+---
+
+## Tool Pre-Fetching
+
+### Memory Tool Configuration
+Enable memory tool pre-fetching to inject saved context into prompts:
+
+```python
+# In your tool configuration
+{
+  "name": "memory",
+  "config": {
+    "pre_fetch_enabled": true  # Default: true
+  }
+}
+```
+
+Control pre-fetching globally:
+```bash
+# .env file
+ENABLE_TOOL_PREFETCH=true
+```
+
+Or per-request:
+```json
+{
+  "question": "What are the requirements?",
+  "disable_tool_prefetch": false
+}
+```
+
+---
+
+## Debugging Prompts
+
+### View Rendered Prompts in Logs
+Set log level to `INFO` to see the final rendered prompt sent to the LLM:
+
+```bash
+export LOG_LEVEL=INFO
+```
+
+You'll see output like:
+```
+INFO - Rendered system prompt for agent (length: 1234 chars):
+================================================================================
+You are a helpful assistant for Acme Corp.
+
+Current date: 2025-10-30
+Request ID: req_abc123
+
+Documents:
+Technical documentation about...
+================================================================================
+```
+
+### Template Validation
+Test your template syntax before saving:
+```python
+from application.api.answer.services.prompt_renderer import PromptRenderer
+
+renderer = PromptRenderer()
+is_valid = renderer.validate_template("Your prompt with {{ variables }}")
+```
+
+---
+
+## Common Use Cases
+
+### 1. Customer Support Bot
+```jinja
+You are a customer support assistant for {{ passthrough.company }}.
+
+**Customer:** {{ passthrough.customer_name }}
+**Ticket ID:** {{ system.request_id }}
+**Date:** {{ system.date }}
+
+**Knowledge Base:**
+{{ source.content }}
+
+**Previous Interactions:**
+{{ tools.memory.root }}
+
+Please provide helpful, friendly support based on the knowledge base above.
+```
+
+### 2. Technical Documentation Assistant
+```jinja
+You are a technical documentation expert.
+
+**Available Documentation ({{ source.count }} documents):**
+{{ source.content }}
+
+**Requirements:**
+- Provide code examples in {{ passthrough.language }}
+- Focus on {{ passthrough.framework }} best practices
+- Include relevant links when possible
+```
+
+### 3. Internal Knowledge Base
+```jinja
+You are an internal AI assistant for {{ passthrough.department }}.
+
+**Employee:** {{ passthrough.employee_name }}
+**Access Level:** {{ passthrough.access_level }}
+
+**Relevant Documents:**
+{{ source.content }}
+
+Provide detailed answers appropriate for {{ passthrough.access_level }} access level.
+```
+
+---
+
+## Template Syntax Reference
+
+### Variables
+```jinja
+{{ variable_name }}              # Output variable
+{{ namespace.field }}            # Access nested field
+{{ variable | default("N/A") }}  # Default value
+```
+
+### Conditionals
+```jinja
+{% if condition %}
+  Content
+{% elif other_condition %}
+  Other content
+{% else %}
+  Default content
+{% endif %}
+```
+
+### Loops
+```jinja
+{% for item in list %}
+  {{ item.field }}
+{% endfor %}
+```
+
+### Comments
+```jinja
+{# This is a comment and won't appear in output #}
+```
+
+---
+
+## Security Considerations
+
+1. **Input Sanitization**: Passthrough data is automatically sanitized to prevent injection attacks
+2. **Type Filtering**: Only primitive types (string, int, float, bool, None) are allowed in passthrough
+3. **Autoescaping**: Jinja2 autoescaping is enabled by default
+4. **Size Limits**: Consider the token budget when including large documents
+
+---
+
+## Troubleshooting
+
+### Problem: Variables Not Rendering
+**Solution:** Ensure you're using the correct namespace:
+```jinja
+❌ {{ company }}
+✅ {{ passthrough.company }}
+```
+
+### Problem: Empty Output for Tool Data
+**Solution:** Check that tool pre-fetching is enabled and the tool is configured correctly.
+
+### Problem: Syntax Errors
+**Solution:** Validate template syntax. Common issues:
+```jinja
+❌ {{ variable }     # Missing closing brace
+❌ {% if x %         # Missing closing %}
+✅ {{ variable }}
+✅ {% if x %}...{% endif %}
+```
+
+### Problem: Legacy Prompts Not Working
+**Solution:** The system auto-detects template syntax. If your prompt uses `{summaries}`, it will work in legacy mode. To use new features, add `{{ }}` syntax.
+
+---
+
+## API Reference
+
+### Render Prompt via API
+```python
+from application.api.answer.services.prompt_renderer import PromptRenderer
+
+renderer = PromptRenderer()
+rendered = renderer.render_prompt(
+    prompt_content="Your template with {{ passthrough.name }}",
+    user_id="user_123",
+    request_id="req_456",
+    passthrough_data={"name": "Alice"},
+    docs_together="Document content here",
+    tools_data={"memory": {"root": "Files: notes.txt"}}
+)
+```
+
+---
+
+## Conclusion
+
+The new template-based prompt system provides powerful flexibility while maintaining backward compatibility. By leveraging namespaces, you can create dynamic, context-aware prompts that adapt to your specific use case.
+
+**Key Benefits:**
+- ✅ Dynamic variable injection
+- ✅ Organized namespaces
+- ✅ Backward compatible
+- ✅ Security built-in
+- ✅ Easy to debug
+
+Start with simple templates and gradually add complexity as needed. Happy prompting! 🚀

docs/content/Guides/How-to-train-on-other-documentation.mdx

@@ -0,0 +1,48 @@
+---
+title: How to Train on Other Documentation
+description: A step-by-step guide on how to effectively train DocsGPT on additional documentation sources.
+---
+
+import { Callout } from 'nextra/components'
+import Image from 'next/image'
+import { Steps } from 'nextra/components'
+
+## How to train on other documentation
+
+Training on other documentation sources can greatly enhance the versatility and depth of DocsGPT's knowledge. By incorporating diverse materials, you can broaden the AI's understanding and improve its ability to generate insightful responses across a range of topics. Here's a step-by-step guide on how to effectively train DocsGPT on additional documentation sources:
+
+**Get your document ready**:
+
+Make sure you have the document on which you want to train on  ready with you on the device which you are using .You can also use links to the documentation to train on.
+
+<Callout type="warning" emoji="⚠️">
+ Note: Supported file formats include .pdf, .txt, .rst, .docx, .md, .mdx, .csv, .epub, .html, .json, .xlsx, .pptx, .png, .jpg, .jpeg, and audio files (.wav, .mp3, .m4a, .ogg, .webm). You can also train using the link of the documentation.
+
+</Callout>
+
+### Video Demo
+
+<Image src="/docs.gif" alt="prompts" width={800} height={500} />
+
+
+
+<Steps>
+### Step1
+Navigate to the sidebar where you will find `Source Docs` option,here you will find 3 options built in which are default,Web Search and None.
+
+
+### Step 2
+Click on the `Upload icon` just beside the source docs options,now browse and upload the document which you want to train on or select the `remote` option if you have to insert the link of the documentation.
+
+
+### Step 3
+Now you will be able to see the name of the file uploaded under the Uploaded Files ,now click on `Train`,once you click on train it might take some time to train on the document. You will be able to see the `Training progress` and once the training is completed you can click the `finish` button and there you go your document is uploaded.
+
+
+### Step 4
+Go to `New chat`  and from the side bar select the document  you uploaded  under the `Source Docs` and go ahead with your chat, now you can ask questions regarding the document you uploaded and you will get the effective answer based on it.
+
+</Steps>
+
+
+

docs/content/Guides/How-to-use-different-LLM.mdx

@@ -0,0 +1,68 @@
+---
+title:
+description:
+---
+
+import { Callout } from 'nextra/components'
+import Image from 'next/image'
+import { Steps } from 'nextra/components'
+
+# Setting Up Local Language Models for Your App
+
+Setting up local language models for your app can significantly enhance its capabilities, enabling it to understand and generate text in multiple languages without relying on external APIs. By integrating local language models, you can improve privacy, reduce latency, and ensure continuous functionality even in offline environments. Here's a comprehensive guide on how to set up local language models for your application:
+
+## Steps:
+### For cloud version LLM change:
+<Steps >
+### Step 1
+Visit the chat screen and you will be to see the default LLM selected.
+### Step 2
+Click on it and you will get a drop down of various LLM's available to choose.
+### Step 3
+Choose the LLM of your choice.
+
+</Steps>
+
+
+
+
+### Video Demo
+<Image src="/llms.gif" alt="prompts" width={800} height={500} />
+
+### For Open source llm change:
+<Steps>
+### Step 1
+For open source version please edit `LLM_PROVIDER`, `LLM_NAME` and others in the .env file. Refer to [⚙️ App Configuration](/Deploying/DocsGPT-Settings) for more information.
+### Step 2
+Visit [☁️ Cloud Providers](/Models/cloud-providers) for the updated list of online models. Make sure you have the right API_KEY and correct LLM_PROVIDER.
+For self-hosted please visit [🖥️ Local Inference](/Models/local-inference).
+</Steps>
+
+## Fallback LLM
+
+DocsGPT can automatically switch to a fallback LLM when the primary model fails, including mid-stream. This works with both streaming and non-streaming requests.
+
+**Fallback order:**
+1. Per-agent backup models (other models configured on the same agent)
+2. Global fallback (`FALLBACK_LLM_*` env vars below)
+3. Error returned if all fail
+
+| Setting | Description | Default |
+| --- | --- | --- |
+| `FALLBACK_LLM_PROVIDER` | Provider name (e.g., `openai`, `anthropic`, `google`) | — |
+| `FALLBACK_LLM_NAME` | Model name (e.g., `gpt-4o`, `claude-sonnet-4-20250514`) | — |
+| `FALLBACK_LLM_API_KEY` | API key for the fallback provider | Falls back to `API_KEY` |
+
+All three (`FALLBACK_LLM_PROVIDER`, `FALLBACK_LLM_NAME`, and an API key) must resolve for the global fallback to activate.
+
+```env
+FALLBACK_LLM_PROVIDER=anthropic
+FALLBACK_LLM_NAME=claude-sonnet-4-20250514
+FALLBACK_LLM_API_KEY=sk-ant-your-anthropic-key
+```
+
+<Callout type="info">
+For maximum resilience, use a fallback provider from a different cloud than your primary. Each agent can also have multiple models configured — the other models are tried first before the global fallback.
+</Callout>
+
+

docs/content/Guides/Integrations/_meta.js

@@ -0,0 +1,18 @@
+export default {
+  "google-drive-connector": {
+    "title": "🔗 Google Drive",
+    "href": "/Guides/Integrations/google-drive-connector"
+  },
+  "sharepoint-connector": {
+    "title": "🔗 SharePoint / OneDrive",
+    "href": "/Guides/Integrations/sharepoint-connector"
+  },
+  "confluence-connector": {
+    "title": "🔗 Confluence",
+    "href": "/Guides/Integrations/confluence-connector"
+  },
+  "mcp-tool-integration": {
+    "title": "🔗 MCP Tools",
+    "href": "/Guides/Integrations/mcp-tool-integration"
+  }
+}

docs/content/Guides/Integrations/confluence-connector.mdx

@@ -0,0 +1,67 @@
+---
+title: Confluence Connector
+description: Connect your Confluence Cloud workspace as an external knowledge base to upload and process pages directly.
+---
+
+import { Callout } from 'nextra/components'
+import { Steps } from 'nextra/components'
+
+# Confluence Connector
+
+Connect your Confluence Cloud workspace to upload and process pages directly as an external knowledge base. Supports page content and attachments (PDFs, Office files, text files, images, and more). Authentication is handled via Atlassian OAuth 2.0 with automatic token refresh.
+
+## Setup
+
+<Steps>
+
+### Step 1: Create an OAuth 2.0 App in Atlassian
+
+1. Go to [developer.atlassian.com/console/myapps](https://developer.atlassian.com/console/myapps/) and click **Create** > **OAuth 2.0 integration**
+2. Under **Authorization**, add a callback URL:
+   - Local: `http://localhost:7091/api/connectors/callback?provider=confluence`
+   - Production: `https://yourdomain.com/api/connectors/callback?provider=confluence`
+
+### Step 2: Configure Permissions
+
+In your app settings, go to **Permissions** and add the **Confluence API**. Enable these scopes:
+- `read:page:confluence`
+- `read:space:confluence`
+- `read:attachment:confluence`
+
+### Step 3: Get Your Credentials
+
+Go to **Settings** in your app to find the **Client ID** and **Secret**. Copy both.
+
+### Step 4: Configure Environment Variables
+
+Add to your backend `.env` file:
+
+```env
+CONFLUENCE_CLIENT_ID=your-atlassian-client-id
+CONFLUENCE_CLIENT_SECRET=your-atlassian-client-secret
+```
+
+Add to your frontend `.env` file:
+
+```env
+VITE_CONFLUENCE_CLIENT_ID=your-atlassian-client-id
+```
+
+| Variable | Description | Required |
+|----------|-------------|----------|
+| `CONFLUENCE_CLIENT_ID` | Client ID from your Atlassian OAuth app | Yes |
+| `CONFLUENCE_CLIENT_SECRET` | Client secret from your Atlassian OAuth app | Yes |
+| `VITE_CONFLUENCE_CLIENT_ID` | Same Client ID, used by the frontend to show the Confluence option | Yes |
+
+### Step 5: Restart and Use
+
+Restart your application, then go to the upload section in DocsGPT and select **Confluence** as the source. You'll be redirected to Atlassian to sign in, then can browse spaces and select pages to process.
+
+</Steps>
+
+## Troubleshooting
+
+- **Option not appearing** — Verify `VITE_CONFLUENCE_CLIENT_ID` is set in the frontend `.env`, then restart.
+- **Authentication failed** — Check that the callback URL matches exactly, including `?provider=confluence`.
+- **No accessible sites** — Ensure the authenticating user has access to at least one Confluence Cloud site.
+- **Permission denied** — Verify that the Confluence API scopes are enabled in your Atlassian app settings.

docs/content/Guides/Integrations/google-drive-connector.mdx

@@ -0,0 +1,73 @@
+---
+title: Google Drive Connector
+description: Connect your Google Drive as an external knowledge base to upload and process files directly from your Google Drive account.
+---
+
+import { Callout } from 'nextra/components'
+import { Steps } from 'nextra/components'
+
+# Google Drive Connector
+
+Connect your Google Drive account to upload and process files directly as an external knowledge base. Supports Google Workspace files (Docs, Sheets, Slides), Office files, PDFs, text files, CSVs, images, and more. Authentication is handled via Google OAuth 2.0 with automatic token refresh.
+
+## Setup
+
+<Steps>
+
+### Step 1: Create a Google Cloud Project
+
+1. Go to the [Google Cloud Console](https://console.cloud.google.com/) and create a new project (or select an existing one)
+2. Navigate to **APIs & Services** > **Library**, search for "Google Drive API", and click **Enable**
+
+### Step 2: Create OAuth 2.0 Credentials
+
+1. Go to **APIs & Services** > **Credentials** > **Create Credentials** > **OAuth client ID**
+2. If prompted, configure the OAuth consent screen (choose **External**, fill in required fields)
+3. Select **Web application** as the application type
+4. Add your DocsGPT URL to **Authorized JavaScript origins** (e.g. `http://localhost:3000`)
+5. Add your callback URL to **Authorized redirect URIs**:
+   - Local: `http://localhost:7091/api/connectors/callback?provider=google_drive`
+   - Production: `https://yourdomain.com/api/connectors/callback?provider=google_drive`
+6. Click **Create** and copy the **Client ID** and **Client Secret**
+
+### Step 3: Configure Environment Variables
+
+Add to your backend `.env` file:
+
+```env
+GOOGLE_CLIENT_ID=your-google-client-id
+GOOGLE_CLIENT_SECRET=your-google-client-secret
+```
+
+Add to your frontend `.env` file:
+
+```env
+VITE_GOOGLE_CLIENT_ID=your-google-client-id
+```
+
+| Variable | Description | Required |
+|----------|-------------|----------|
+| `GOOGLE_CLIENT_ID` | OAuth Client ID from GCP Credentials | Yes |
+| `GOOGLE_CLIENT_SECRET` | OAuth Client Secret from GCP Credentials | Yes |
+| `VITE_GOOGLE_CLIENT_ID` | Same Client ID, used by the frontend to show the Google Drive option | Yes |
+
+<Callout type="warning" emoji="⚠️">
+Make sure to use the same Google Client ID in both backend and frontend configurations.
+</Callout>
+
+### Step 4: Restart and Use
+
+Restart your application, then go to the upload section in DocsGPT and select **Google Drive** as the source. You'll be redirected to Google to sign in, then can browse and select files to process.
+
+</Steps>
+
+## Troubleshooting
+
+- **Option not appearing** — Verify `VITE_GOOGLE_CLIENT_ID` is set in the frontend `.env`, then restart.
+- **Authentication failed** — Check that the redirect URI matches exactly, including `?provider=google_drive`. Ensure the Google Drive API is enabled.
+- **Permission denied** — Verify the OAuth consent screen is configured and the user has access to the target files.
+- **Files not processing** — Check backend logs and verify that backend environment variables are correctly set.
+
+<Callout type="tip" emoji="💡">
+For production deployments, add your actual domain to the OAuth consent screen and authorized origins/redirect URIs.
+</Callout>

docs/content/Guides/Integrations/mcp-tool-integration.mdx

@@ -0,0 +1,66 @@
+---
+title: MCP Tool Integration
+description: Connect external tools to DocsGPT agents using the Model Context Protocol (MCP) standard.
+---
+
+import { Callout } from 'nextra/components'
+import { Steps } from 'nextra/components'
+
+# MCP Tool Integration
+
+The [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) integration lets you connect external tool servers to DocsGPT. Your agents can then discover and call tools provided by those servers during conversations — for example, querying a CRM, running code, or accessing a database.
+
+## Setup
+
+<Steps>
+
+### Step 1: Configure Environment Variables (Optional)
+
+Only needed if your MCP servers use OAuth authentication:
+
+```env
+MCP_OAUTH_REDIRECT_URI=https://yourdomain.com/api/mcp_server/callback
+```
+
+If not set, falls back to `API_URL/api/mcp_server/callback`.
+
+### Step 2: Add an MCP Server
+
+Go to **Settings** > **Tools** > **Add Tool** > **MCP Server**. Enter the server URL, select an auth type, and click **Test Connection** to verify, then **Save**.
+
+### Step 3: Enable for Your Agent
+
+In your agent configuration, enable the MCP tools you want the agent to use.
+
+</Steps>
+
+## Authentication Types
+
+| Auth Type | Config Fields |
+|-----------|---------------|
+| **None** | — |
+| **Bearer** | `bearer_token` |
+| **API Key** | `api_key`, `api_key_header` (default: `X-API-Key`) |
+| **Basic** | `username`, `password` |
+| **OAuth** | `oauth_scopes` (optional) |
+
+<Callout type="warning">
+For OAuth in production, `MCP_OAUTH_REDIRECT_URI` must be a publicly accessible URL pointing to your DocsGPT backend.
+</Callout>
+
+## API Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/mcp_server/test` | POST | Test a connection without saving |
+| `/api/mcp_server/save` | POST | Save or update a server configuration |
+| `/api/mcp_server/callback` | GET | OAuth callback handler |
+| `/api/mcp_server/oauth_status/<task_id>` | GET | Poll OAuth flow status |
+| `/api/mcp_server/auth_status` | GET | Batch check auth status for all MCP tools |
+
+## Troubleshooting
+
+- **Connection refused** — Verify the URL and that the server is reachable from your backend.
+- **403 Forbidden** — Check credentials and permissions.
+- **Timed out** — Default is 30s; increase timeout in tool config (max 300s).
+- **OAuth "needs_auth" persists** — Verify `MCP_OAUTH_REDIRECT_URI` is correct and Redis is running.

docs/content/Guides/Integrations/sharepoint-connector.mdx

@@ -0,0 +1,63 @@
+---
+title: SharePoint / OneDrive Connector
+description: Connect your Microsoft SharePoint or OneDrive as an external knowledge base to upload and process files directly.
+---
+
+import { Callout } from 'nextra/components'
+import { Steps } from 'nextra/components'
+
+# SharePoint / OneDrive Connector
+
+Connect your SharePoint or OneDrive account to upload and process files directly as an external knowledge base. Supports Office files, PDFs, text files, CSVs, images, and more. Authentication is handled via Microsoft Entra ID (Azure AD) with automatic token refresh.
+
+## Setup
+
+<Steps>
+
+### Step 1: Create an App Registration in Azure
+
+1. Go to the [Azure Portal](https://portal.azure.com/) > **Microsoft Entra ID** > **App registrations** > **New registration**
+2. Set **Redirect URI** (Web) to:
+   - Local: `http://localhost:7091/api/connectors/callback?provider=share_point`
+   - Production: `https://yourdomain.com/api/connectors/callback?provider=share_point`
+
+### Step 2: Configure API Permissions
+
+In your App Registration, go to **API permissions** > **Add a permission** > **Microsoft Graph** > **Delegated permissions** and add: `Files.Read`, `Files.Read.All`, `Sites.Read.All`. Grant admin consent if possible.
+
+### Step 3: Create a Client Secret
+
+Go to **Certificates & secrets** > **New client secret**. Copy the secret value immediately (it won't be shown again).
+
+### Step 4: Configure Environment Variables
+
+Add to your `.env` file:
+
+```env
+MICROSOFT_CLIENT_ID=your-azure-ad-client-id
+MICROSOFT_CLIENT_SECRET=your-azure-ad-client-secret
+MICROSOFT_TENANT_ID=your-azure-ad-tenant-id
+```
+
+| Variable | Description | Required | Default |
+|----------|-------------|----------|---------|
+| `MICROSOFT_CLIENT_ID` | Application (client) ID from App Registration overview | Yes | — |
+| `MICROSOFT_CLIENT_SECRET` | Client secret value | Yes | — |
+| `MICROSOFT_TENANT_ID` | Directory (tenant) ID | No | `common` |
+| `MICROSOFT_AUTHORITY` | Login endpoint override | No | Auto-constructed |
+
+<Callout type="warning">
+`MICROSOFT_TENANT_ID=common` (the default) allows any Microsoft account to authenticate. Set this to your specific tenant ID in production.
+</Callout>
+
+### Step 5: Restart and Use
+
+Restart your application, then go to the upload section in DocsGPT and select **SharePoint / OneDrive** as the source. You'll be redirected to Microsoft to sign in, then can browse and select files to process.
+
+</Steps>
+
+## Troubleshooting
+
+- **Option not appearing** — Verify `MICROSOFT_CLIENT_ID` and `MICROSOFT_CLIENT_SECRET` are set, then restart.
+- **Authentication failed** — Check that the redirect URI matches exactly, including `?provider=share_point`.
+- **Permission denied** — Ensure admin consent is granted and the user has access to the target files.

docs/content/Guides/My-AI-answers-questions-using-external-knowledge.mdx

@@ -0,0 +1,16 @@
+---
+title:
+description:
+---
+
+# Avoiding hallucinations
+
+If your AI uses external knowledge and is not explicit enough, it is ok, because we try to make DocsGPT friendly.
+
+But if you want to adjust it, prompts are now managed through the UI and API using a template-based system. See the [Customising Prompts](/Guides/Customising-prompts) guide for details.
+
+To make the AI stricter about staying on-topic, edit your active prompt template (via **Sidebar → Settings → Active Prompt**) to include instructions like:
+
+```
+If the context provides insufficient information, reply "I cannot answer".
+```

docs/content/Guides/_meta.js

@@ -0,0 +1,35 @@
+export default {
+  "Customising-prompts": {
+    "title": "️💻 Customising Prompts",
+    "href": "/Guides/Customising-prompts"
+  },
+  "How-to-train-on-other-documentation": {
+    "title": "📥 Training on docs",
+    "href": "/Guides/How-to-train-on-other-documentation"
+  },
+  "How-to-use-different-LLM": {
+    "title": "️🤖 How to use different LLM's",
+    "href": "/Guides/How-to-use-different-LLM",
+    "display": "hidden"
+  },
+  "My-AI-answers-questions-using-external-knowledge": {
+    "title": "💭️ Avoiding hallucinations",
+    "href": "/Guides/My-AI-answers-questions-using-external-knowledge",
+    "display": "hidden"
+  },
+  "Architecture": {
+    "title": "🏗️ Architecture",
+    "href": "/Guides/Architecture"
+  },
+  "compression": {
+    "title": "🗜️ Context Compression",
+    "href": "/Guides/compression"
+  },
+  "ocr": {
+    "title": "OCR",
+    "href": "/Guides/ocr"
+  },
+"Integrations": {
+    "title": "🔗 Integrations"
+  }
+}

docs/content/Guides/compression.md

@@ -0,0 +1,37 @@
+# Context Compression
+
+DocsGPT implements a smart context compression system to manage long conversations effectively. This feature prevents conversations from hitting the LLM's context window limit while preserving critical information and continuity.
+
+## How It Works
+
+The compression system operates on a "summarize and truncate" principle:
+
+1.  **Threshold Check**: Before each request, the system calculates the total token count of the conversation history.
+2.  **Trigger**: If the token count exceeds a configured threshold (default: 80% of the model's context limit), compression is triggered.
+3.  **Summarization**: An LLM (potentially a different, cheaper/faster one) processes the older part of the conversation—including previous summaries, user messages, agent responses, and tool outputs.
+4.  **Context Replacement**: The system generates a comprehensive summary of the older history. For subsequent requests, the LLM receives this **Summary + Recent Messages** instead of the full raw history.
+
+### Key Features
+
+*   **Recursive Summarization**: New summaries incorporate previous summaries, ensuring that information from the very beginning of a long chat is not lost.
+*   **Tool Call Support**: The compression logic explicitly handles tool calls and their outputs (e.g., file readings, search results), summarizing their results so the agent retains knowledge of what it has already done.
+*   **"Needle in a Haystack" Preservation**: The prompts are designed to identify and preserve specific, critical details (like passwords, keys, or specific user instructions) even when compressing large amounts of text.
+
+## Configuration
+
+You can configure the compression behavior in your `.env` file or `application/core/settings.py`:
+
+| Setting | Default | Description |
+| :--- | :--- | :--- |
+| `ENABLE_CONVERSATION_COMPRESSION` | `True` | Master switch to enable/disable the feature. |
+| `COMPRESSION_THRESHOLD_PERCENTAGE` | `0.8` | The fraction of the context window (0.0 to 1.0) that triggers compression. |
+| `COMPRESSION_MODEL_OVERRIDE` | `None` | (Optional) Specify a different model ID to use specifically for the summarization task (e.g., using `gpt-3.5-turbo` to compress for `gpt-4`). |
+| `COMPRESSION_MAX_HISTORY_POINTS` | `3` | The number of past compression points to keep in the database (older ones are discarded as they are incorporated into newer summaries). |
+
+## Architecture
+
+The system is modularized into several components:
+
+*   **`CompressionThresholdChecker`**: Calculates token usage and decides when to compress.
+*   **`CompressionService`**: Orchestrates the compression process, manages DB updates, and reconstructs the context (Summary + Recent Messages) for the LLM.
+*   **`CompressionPromptBuilder`**: Constructs the specific prompts used to instruct the LLM to summarize the conversation effectively.

docs/content/Guides/ocr.mdx

@@ -0,0 +1,80 @@
+---
+title: OCR for Sources and Attachments
+description: How OCR works in DocsGPT, how to configure it, and what changes for source ingestion vs chat attachments.
+---
+
+import { Callout } from 'nextra/components'
+
+# Docling OCR for Sources and Attachments
+
+DocsGPT uses Docling as the default parser layer for many document formats. OCR is optional and controlled by two settings:
+
+```env
+DOCLING_OCR_ENABLED=false
+DOCLING_OCR_ATTACHMENTS_ENABLED=false
+```
+
+- `DOCLING_OCR_ENABLED`: OCR behavior for Source Docs ingestion.
+- `DOCLING_OCR_ATTACHMENTS_ENABLED`: OCR behavior for chat attachments uploaded from the message box.
+
+## Processing Flow
+
+### Source Docs flow (Upload and Train)
+
+1. Files are uploaded through `/api/upload`.
+2. Ingestion runs asynchronously in Celery (`ingest_worker`).
+3. `SimpleDirectoryReader` parses files with `get_default_file_extractor`.
+4. For PDFs and image formats, Docling parsers are used. OCR in this path is controlled by `DOCLING_OCR_ENABLED`.
+5. Parsed text is chunked, embedded, and stored in the vector store.
+6. Retrieval during chat uses this indexed text and returns source citations.
+
+### Attachment flow (Chat-only file context)
+
+1. Files are uploaded through `/api/store_attachment`.
+2. Celery task `attachment_worker` parses and stores the attachment in Postgres (`attachments` table).
+3. OCR in this path is controlled by `DOCLING_OCR_ATTACHMENTS_ENABLED`.
+4. Attachments are not vectorized and are not added to the source index.
+5. During answer generation, selected attachment IDs are loaded and passed directly to the LLM pipeline.
+
+## How Docling OCR Works
+
+Docling OCR behavior is different for PDFs vs images:
+
+- PDF parser defaults to hybrid OCR:
+  - text regions: extracted directly
+  - bitmap/image regions: OCR only where needed
+- Image parser defaults to full-page OCR (the whole image is visual content).
+
+By default, Docling parser classes use RapidOCR options (language default: `english`).
+
+<Callout type="info" emoji="ℹ️">
+Parser internals like OCR language and force-full-page OCR are currently set by code defaults, not separate `.env` settings.
+</Callout>
+
+## Attachment Behavior by Model Support
+
+When attachments are used in chat, behavior depends on the selected model/provider:
+
+- If a MIME type is supported, DocsGPT sends files/images through provider-native attachment APIs.
+- If unsupported, DocsGPT falls back to the parsed text content stored for the attachment.
+- For providers that support images but not native PDF attachments, PDF files are converted to images (synthetic PDF support).
+
+This means OCR quality is especially important for text fallback paths and for models without native attachment support.
+
+## Recommended Configuration
+
+For most OCR-enabled use cases, enable both flags:
+
+```env
+DOCLING_OCR_ENABLED=true
+DOCLING_OCR_ATTACHMENTS_ENABLED=true
+```
+
+After changing these settings, restart the API and Celery worker.
+
+## Legacy Fallback Notes
+
+- If Docling is unavailable, DocsGPT falls back to legacy parsers.
+- With OCR disabled, text-based PDFs can still parse, but scanned/image-heavy content may produce little text.
+- For image parsing without Docling OCR, the legacy image parser only extracts text when `PARSE_IMAGE_REMOTE=true`.
+