chore: import upstream snapshot with attribution

docs/README.md

@@ -0,0 +1,51 @@
+# nextra-docsgpt
+
+## Setting Up Docs Folder of DocsGPT Locally
+
+### 1. Clone the DocsGPT repository:
+
+```bash
+git clone https://github.com/arc53/DocsGPT.git
+```
+### 2. Navigate to the docs folder:
+
+```bash
+cd DocsGPT/docs
+```
+
+The docs folder contains the markdown files that make up the documentation. The majority of the files are in the pages directory. Some notable files in this folder include:
+
+`index.mdx`: The main documentation file.
+`_app.js`: This file is used to customize the default Next.js application shell.
+`theme.config.jsx`: This file is for configuring the Nextra theme for the documentation.
+
+### 3. Verify that you have Node.js and npm installed in your system. You can check by running:
+
+```bash
+node --version
+npm --version
+```
+
+### 4. If not installed, download Node.js and npm from the respective official websites.
+
+### 5. Once you have Node.js and npm running, proceed to install yarn - another package manager that helps to manage project dependencies:
+
+```bash
+npm install --global yarn
+```
+
+### 6. Install the project dependencies using yarn:
+
+```bash
+yarn install
+```
+
+### 7. After the successful installation of the project dependencies, start the local server:
+
+```bash
+yarn dev
+```
+
+- Now, you should be able to view the docs on your local environment by visiting `http://localhost:3000`. You can explore the different markdown files and make changes as you see fit.
+
+- **Footnotes:** This guide assumes you have Node.js and npm installed. The guide involves running a local server using yarn, and viewing the documentation offline. If you encounter any issues, it may be worth verifying your Node.js and npm installations and whether you have installed yarn correctly.

docs/app/[[...mdxPath]]/page.jsx

@@ -0,0 +1,25 @@
+import { generateStaticParamsFor, importPage } from 'nextra/pages';
+
+import { useMDXComponents } from '../../mdx-components';
+
+export const generateStaticParams = generateStaticParamsFor('mdxPath');
+
+export async function generateMetadata(props) {
+  const params = await props.params;
+  const { metadata } = await importPage(params?.mdxPath);
+  return metadata;
+}
+
+const Wrapper = useMDXComponents().wrapper;
+
+export default async function Page(props) {
+  const params = await props.params;
+  const result = await importPage(params?.mdxPath);
+  const { default: MDXContent, metadata, sourceCode, toc } = result;
+
+  return (
+    <Wrapper metadata={metadata} sourceCode={sourceCode} toc={toc}>
+      <MDXContent {...props} params={params} />
+    </Wrapper>
+  );
+}

docs/app/layout.jsx

@@ -0,0 +1,86 @@
+import Image from 'next/image';
+import { Analytics } from '@vercel/analytics/react';
+import { Banner, Head } from 'nextra/components';
+import { getPageMap } from 'nextra/page-map';
+import { Footer, Layout, Navbar } from 'nextra-theme-docs';
+import 'nextra-theme-docs/style.css';
+
+import CuteLogo from '../public/cute-docsgpt.png';
+import themeConfig from '../theme.config';
+
+const github = 'https://github.com/arc53/DocsGPT';
+
+export const metadata = {
+  title: {
+    default: 'DocsGPT Documentation',
+    template: '%s - DocsGPT Documentation',
+  },
+  description:
+    'Use DocsGPT to chat with your data. DocsGPT is a GPT-powered chatbot that can answer questions about your data.',
+};
+
+const navbar = (
+  <Navbar
+    logo={
+      <div style={{ alignItems: 'center', display: 'flex', gap: '8px' }}>
+        <Image src={CuteLogo} alt="DocsGPT logo" width={28} height={28} />
+        <span style={{ fontWeight: 'bold', fontSize: 18 }}>DocsGPT Docs</span>
+      </div>
+    }
+    projectLink={github}
+    chatLink="https://discord.com/invite/n5BX8dh8rU"
+  />
+);
+
+const footer = (
+  <Footer>
+    <span>MIT {new Date().getFullYear()} © </span>
+    <a href="https://www.docsgpt.cloud/" target="_blank" rel="noreferrer">
+      DocsGPT
+    </a>
+    {' | '}
+    <a href="https://github.com/arc53/DocsGPT" target="_blank" rel="noreferrer">
+      GitHub
+    </a>
+    {' | '}
+    <a href="https://blog.docsgpt.cloud/" target="_blank" rel="noreferrer">
+      Blog
+    </a>
+  </Footer>
+);
+
+export default async function RootLayout({ children }) {
+  return (
+    <html lang="en" dir="ltr" suppressHydrationWarning>
+      <Head>
+        <link
+          rel="apple-touch-icon"
+          sizes="180x180"
+          href="/favicons/apple-touch-icon.png"
+        />
+        <link rel="icon" type="image/png" sizes="32x32" href="/favicons/favicon-32x32.png" />
+        <link rel="icon" type="image/png" sizes="16x16" href="/favicons/favicon-16x16.png" />
+        <link rel="manifest" href="/favicons/site.webmanifest" />
+        <meta httpEquiv="Content-Language" content="en" />
+      </Head>
+      <body>
+        <Layout
+          banner={
+            <Banner storageKey="docs-launch">
+              <div className="flex justify-center items-center gap-2">
+                Welcome to the new DocsGPT docs!
+              </div>
+            </Banner>
+          }
+          navbar={navbar}
+          footer={footer}
+          pageMap={await getPageMap()}
+          {...themeConfig}
+        >
+          {children}
+        </Layout>
+        <Analytics />
+      </body>
+    </html>
+  );
+}

docs/components/DeploymentCards.jsx

@@ -0,0 +1,122 @@
+'use client';
+
+import Image from 'next/image';
+
+const iconMap = {
+    'Amazon Lightsail': '/lightsail.png',
+    'Railway': '/railway.png',
+    'Civo Compute Cloud': '/civo.png',
+    'DigitalOcean Droplet': '/digitalocean.png',
+    'Kamatera Cloud': '/kamatera.png',
+};
+
+
+export function DeploymentCards({ items }) {
+    return (
+        <>
+            <div className="deployment-cards">
+                {items.map(({ title, link, description }) => {
+                    const isExternal = link.startsWith('https://');
+                    const iconSrc = iconMap[title] || '/default-icon.png'; // Default icon if not found
+
+                    return (
+                        <div
+                            key={title}
+                            className={`card${isExternal ? ' external' : ''}`}
+                        >
+                            <a href={link} target={isExternal ? '_blank' : undefined} rel="noopener noreferrer" className="card-link-wrapper">
+                                <div className="card-icon-container">
+                                    {iconSrc && <div className="card-icon"><Image src={iconSrc} alt={title} width={32} height={32} /></div>} {/* Reduced icon size */}
+                                </div>
+                                <h3 className="card-title">{title}</h3>
+                                {description && <p className="card-description">{description}</p>}
+                                <p className="card-url">{new URL(link).hostname.replace('www.', '')}</p>
+                            </a>
+                        </div>
+                    );
+                })}
+            </div>
+
+            <style jsx>{`
+               .deployment-cards {
+                    margin-top: 24px;
+                    display: grid;
+                    grid-template-columns: 1fr;
+                    gap: 16px;
+                }
+                @media (min-width: 768px) {
+                    .deployment-cards {
+                        grid-template-columns: 1fr 1fr;
+                    }
+                }
+                .card {
+                    background-color: #222222;
+                    border-radius: 8px;
+                    padding: 16px;
+                    transition: background-color 0.3s;
+                    position: relative;
+                    color: #ffffff;
+                    /* Make the card a flex container */
+                    display: flex;
+                    flex-direction: column;
+                    align-items: center; /* Center horizontally */
+                    justify-content: center; /* Center vertically */
+                    height: 100%; /* Fill the height of the grid cell */
+                  
+                }
+                .card:hover {
+                    background-color: #333333;
+                }
+                .card.external::after {
+                    content: "↗";
+                    position: absolute;
+                    top: 12px; /* Adjusted position */
+                    right: 12px; /* Adjusted position */
+                    color: #ffffff;
+                    font-size: 0.7em; /* Reduced size */
+                    opacity: 0.8; /* Slightly faded */
+                }
+                .card-link-wrapper {
+                    display: flex;
+                    flex-direction: column;
+                    align-items:center;
+                    color: inherit;
+                    text-decoration: none;
+                    width:100%; /* Important: make link wrapper take full width */
+                }
+               .card-icon-container{
+                display:flex;
+                justify-content:center;
+                 width: 100%;
+                 margin-bottom: 8px; /* Space between icon and title */
+               }
+                .card-icon {
+                   display: block;
+                   margin: 0 auto;
+
+                }
+                .card-title {
+                    font-weight: 600;
+                    margin-bottom: 4px;
+                    font-size: 16px;
+                    text-align: center;
+                    color: #f0f0f0; /* Lighter title color if needed */
+                }
+                .card-description {
+                    margin-bottom: 0;
+                    font-size: 13px;
+                    color: #aaaaaa;
+                    text-align: center;
+                    line-height: 1.4;
+                }
+                .card-url {
+                    margin-top: 8px; /*Keep space consistent */
+                    font-size: 11px;
+                    color: #777777;
+                    text-align: center;
+                    font-family: monospace;
+                }
+            `}</style>
+        </>
+    );
+}

docs/components/ToolCards.jsx

@@ -0,0 +1,125 @@
+'use client';
+
+import Image from 'next/image';
+
+const iconMap = {
+    'API Tool': '/toolIcons/tool_api_tool.svg',
+    'Brave Search': '/toolIcons/tool_brave.svg',
+    'DuckDuckGo Search': '/toolIcons/tool_duckduckgo.svg',
+    'CryptoPrice': '/toolIcons/tool_cryptoprice.svg',
+    'Ntfy': '/toolIcons/tool_ntfy.svg',
+    'Telegram Bot': '/toolIcons/tool_telegram.svg',
+    'PostgreSQL Database': '/toolIcons/tool_postgres.svg',
+    'Read Webpage (browser)': '/toolIcons/tool_read_webpage.svg',
+    'Remote Device': '/toolIcons/tool_remote_device.svg',
+    'MCP Tool': '/toolIcons/tool_mcp_tool.svg',
+    'Memory': '/toolIcons/tool_memory.svg',
+    'Notepad': '/toolIcons/tool_notes.svg',
+    'Todo List': '/toolIcons/tool_todo_list.svg'
+};
+
+
+export function ToolCards({ items }) {
+    return (
+        <>
+            <div className="tool-cards">
+                {items.map(({ title, link, description }) => {
+                    const isExternal = link.startsWith('https://');
+                    const iconSrc = iconMap[title]; // No icon rendered when the tool has none
+
+                    return (
+                        <div
+                            key={title}
+                            className={`card${isExternal ? ' external' : ''}`}
+                        >
+                            <a href={link} target={isExternal ? '_blank' : undefined} rel="noopener noreferrer" className="card-link-wrapper">
+                                <div className="card-icon-container">
+                                    {iconSrc && <div className="card-icon"><Image src={iconSrc} alt={title} width={32} height={32} /></div>} {/* Reduced icon size */}
+                                </div>
+                                <h3 className="card-title">{title}</h3>
+                                {description && <p className="card-description">{description}</p>}
+                                {/* Card URL element removed from here */}
+                            </a>
+                        </div>
+                    );
+                })}
+            </div>
+
+            <style jsx>{`
+               .tool-cards {
+                    margin-top: 24px;
+                    display: grid;
+                    grid-template-columns: 1fr;
+                    gap: 16px;
+                }
+                @media (min-width: 768px) {
+                    .tool-cards {
+                        grid-template-columns: 1fr 1fr; /* Keeps two columns on wider screens */
+                    }
+                }
+                .card {
+                    background-color: #222222;
+                    border-radius: 8px;
+                    padding: 16px; /* Existing padding */
+                    transition: background-color 0.3s;
+                    position: relative;
+                    color: #ffffff;
+                    display: flex; /* Using flex to help with alignment */
+                    flex-direction: column;
+                    /* align-items: center; // Alignment for items inside card-link-wrapper is better */
+                    /* justify-content: center; // We want content to flow from top */
+                    height: 100%; /* Fill the height of the grid cell, ensures cards in a row are same height */
+                }
+                .card:hover {
+                    background-color: #333333;
+                }
+                .card.external::after {
+                    content: "↗";
+                    position: absolute;
+                    top: 12px;
+                    right: 12px;
+                    color: #ffffff;
+                    font-size: 0.7em;
+                    opacity: 0.8;
+                }
+                .card-link-wrapper {
+                    display: flex;
+                    flex-direction: column;
+                    align-items:center; /* Centers icon, title, description horizontally */
+                    text-align: center; /* Ensures text within p and h3 is centered */
+                    color: inherit;
+                    text-decoration: none;
+                    width:100%;
+                    height: 100%; /* Make the link wrapper take full card height */
+                    justify-content: flex-start; /* Align content to the top */
+                }
+               .card-icon-container{
+                    display:flex;
+                    justify-content:center;
+                    width: 100%;
+                    margin-top: 8px; /* Added some margin at the top if needed */
+                    margin-bottom: 12px; /* Increased space between icon and title */
+               }
+                .card-icon {
+                   display: block;
+                   /* margin: 0 auto; // Center handled by card-icon-container */
+                }
+                .card-title {
+                    font-weight: 600;
+                    margin-bottom: 8px; /* Increased space below title */
+                    font-size: 16px; /* Consider increasing slightly if descriptions are longer e.g. 17px or 18px */
+                    color: #f0f0f0;
+                }
+                .card-description {
+                    /* margin-bottom: 0; // Original value */
+                    font-size: 14px; /* Slightly increased font size for better readability */
+                    color: #aaaaaa;
+                    line-height: 1.5; /* Slightly increased line height */
+                    flex-grow: 1; /* Allows description to take available space */
+                    overflow-y: auto; /* Adds scroll if description is too long, though ideally content fits */
+                    padding-bottom: 8px; /* Add some padding at the bottom of the description area */
+                }
+            `}</style>
+        </>
+    );
+}

docs/content/Agents/_meta.js

@@ -0,0 +1,26 @@
+export default {
+  "basics": {
+    "title": "🤖 Agent Basics",
+    "href": "/Agents/basics"
+  },
+  "api": {
+    "title": "🔌 Agent API",
+    "href": "/Agents/api"
+  },
+  "openai-compatible": {
+    "title": "🔄 OpenAI-Compatible API",
+    "href": "/Agents/openai-compatible"
+  },
+  "webhooks": {
+    "title": "🪝 Agent Webhooks",
+    "href": "/Agents/webhooks"
+  },
+  "notifications": {
+    "title": "📡 Realtime Events",
+    "href": "/Agents/notifications"
+  },
+  "nodes": {
+    "title": "🧩 Workflow Nodes",
+    "href": "/Agents/nodes"
+  }
+}

docs/content/Agents/api.mdx

@@ -0,0 +1,395 @@
+---
+title: Interacting with Agents via API
+description: Learn how to programmatically interact with DocsGPT Agents using the streaming and non-streaming API endpoints.
+---
+
+import { Callout, Tabs } from 'nextra/components';
+
+# Interacting with Agents via API
+
+DocsGPT Agents can be accessed programmatically through API endpoints. This page covers:
+
+- Non-streaming answers (`/api/answer`)
+- Streaming answers over SSE (`/stream`)
+- File/image attachments (`/api/store_attachment` + `/api/task_status` + `/stream`)
+
+When you use an agent `api_key`, DocsGPT loads that agent's configuration automatically (prompt, tools, sources, default model). You usually only need to send `question` and `api_key`.
+
+<Callout type="info">
+Looking to connect an existing OpenAI-compatible client (opencode, aider, the OpenAI SDKs, etc.) to a DocsGPT Agent? Use the [OpenAI-Compatible Chat Completions API](/Agents/openai-compatible) — it speaks the standard chat completions protocol so no adapter code is required.
+</Callout>
+
+## Base URL
+
+<Callout type="info">
+For DocsGPT Cloud, use `https://gptcloud.arc53.com` as the base URL.
+</Callout>
+
+- Local: `http://localhost:7091`
+- Cloud: `https://gptcloud.arc53.com`
+
+## How Request Resolution Works
+
+DocsGPT resolves your request in this order:
+
+1. If `api_key` is provided, DocsGPT loads the mapped agent and executes with that config.
+2. If `agent_id` is provided (typically with JWT auth), DocsGPT loads that agent if allowed.
+3. If neither is provided, DocsGPT uses request-level fields (`prompt_id`, `active_docs`, `retriever`, etc.).
+
+Authentication:
+
+- Agent API-key flow: include `api_key` in JSON/form payload.
+- JWT flow (if auth enabled): include `Authorization: Bearer <token>`.
+
+## Endpoints
+
+- `POST /api/answer` (non-streaming)
+- `POST /stream` (SSE streaming)
+- `POST /api/store_attachment` (multipart upload)
+- `GET /api/task_status?task_id=...` (Celery task polling)
+
+## Request Parameters
+
+Common request body fields:
+
+| Field | Type | Required | Applies to | Notes |
+| --- | --- | --- | --- | --- |
+| `question` | `string` | Yes | `/api/answer`, `/stream` | User query. |
+| `api_key` | `string` | Usually | `/api/answer`, `/stream` | Recommended for agent API use. Loads agent config from key. |
+| `conversation_id` | `string` | No | `/api/answer`, `/stream` | Continue an existing conversation. |
+| `history` | `string` (JSON-encoded array) | No | `/api/answer`, `/stream` | Used for new conversations. Format: `[{\"prompt\":\"...\",\"response\":\"...\"}]`. |
+| `model_id` | `string` | No | `/api/answer`, `/stream` | Override model for this request. |
+| `save_conversation` | `boolean` | No | `/api/answer`, `/stream` | Deprecated, no effect. Conversations are always persisted; use `visibility` to control sidebar listing. |
+| `visibility` | `string` | No | `/api/answer`, `/stream` | `"listed"` shows the conversation in the owner's sidebar. Any other value (or omitting it) persists it hidden. Default `hidden`. |
+| `passthrough` | `object` | No | `/api/answer`, `/stream` | Dynamic values injected into prompt templates. |
+| `prompt_id` | `string` | No | `/api/answer`, `/stream` | Ignored when `api_key` already defines prompt. |
+| `active_docs` | `string` or `string[]` | No | `/api/answer`, `/stream` | Overrides active docs when not using key-owned source config. |
+| `retriever` | `string` | No | `/api/answer`, `/stream` | Retriever type (for example `classic`). |
+| `chunks` | `number` | No | `/api/answer`, `/stream` | Retrieval chunk count, default `2`. |
+| `isNoneDoc` | `boolean` | No | `/api/answer`, `/stream` | Skip document retrieval. |
+| `agent_id` | `string` | No | `/api/answer`, `/stream` | Alternative to `api_key` when using authenticated user context. |
+
+Streaming-only fields:
+
+| Field | Type | Required | Notes |
+| --- | --- | --- | --- |
+| `attachments` | `string[]` | No | List of attachment IDs from `/api/task_status` success result. |
+| `index` | `number` | No | Update an existing query index. If provided, `conversation_id` is required. |
+
+## Non-Streaming API (`/api/answer`)
+
+`/api/answer` waits for completion and returns one JSON response.
+
+<Callout type="info">
+`attachments` are currently handled through `/stream`. For file/image-attached queries, use the streaming endpoint.
+</Callout>
+
+Response fields:
+
+- `conversation_id`
+- `answer`
+- `sources`
+- `tool_calls`
+- `thought`
+- Optional structured output metadata (`structured`, `schema`) when enabled
+
+### Examples
+
+<Tabs items={['cURL', 'Python', 'JavaScript']}>
+  <Tabs.Tab>
+    ```bash
+    curl -X POST http://localhost:7091/api/answer \
+    -H "Content-Type: application/json" \
+    -d '{"question":"your question here","api_key":"your_agent_api_key"}'
+    ```
+  </Tabs.Tab>
+  <Tabs.Tab>
+    ```python
+    import requests
+
+    API_URL = "http://localhost:7091/api/answer"
+    API_KEY = "your_agent_api_key"
+    QUESTION = "your question here"
+
+    response = requests.post(
+        API_URL,
+        json={"question": QUESTION, "api_key": API_KEY}
+    )
+
+    if response.status_code == 200:
+        print(response.json())
+    else:
+        print(f"Error: {response.status_code}")
+        print(response.text)
+    ```
+  </Tabs.Tab>
+  <Tabs.Tab>
+    ```javascript
+    const apiUrl = 'http://localhost:7091/api/answer';
+    const apiKey = 'your_agent_api_key';
+    const question = 'your question here';
+
+    async function getAnswer() {
+    try {
+        const response = await fetch(apiUrl, {
+        method: 'POST',
+        headers: {
+            'Content-Type': 'application/json',
+        },
+        body: JSON.stringify({ question, api_key: apiKey }),
+        });
+
+        if (!response.ok) {
+        throw new Error(`HTTP error! Status: ${response.status}`);
+        }
+
+        const data = await response.json();
+        console.log(data);
+    } catch (error) {
+        console.error("Failed to fetch answer:", error);
+    }
+    }
+
+    getAnswer();
+    ```
+  </Tabs.Tab>
+</Tabs>
+
+---
+
+## Streaming API (`/stream`)
+
+`/stream` returns a Server-Sent Events (SSE) stream so you can render output token-by-token.
+
+### SSE Event Types
+
+Each `data:` frame is JSON with `type`:
+
+- `answer`: incremental answer chunk
+- `source`: source list/chunks
+- `tool_calls`: tool invocation results/metadata
+- `thought`: reasoning/thought chunk (agent dependent)
+- `structured_answer`: final structured payload (when schema mode is active)
+- `id`: final conversation ID
+- `error`: error message
+- `end`: stream is complete
+
+### Examples
+
+<Tabs items={['cURL', 'Python', 'JavaScript']}>
+  <Tabs.Tab>
+    ```bash
+    curl -X POST http://localhost:7091/stream \
+    -H "Content-Type: application/json" \
+    -H "Accept: text/event-stream" \
+    -d '{"question":"your question here","api_key":"your_agent_api_key"}'
+    ```
+  </Tabs.Tab>
+  <Tabs.Tab>
+    ```python
+    import requests
+    import json
+
+    API_URL = "http://localhost:7091/stream"
+    payload = {
+        "question": "your question here",
+        "api_key": "your_agent_api_key"
+    }
+
+    with requests.post(API_URL, json=payload, stream=True) as r:
+        for line in r.iter_lines():
+            if line:
+                decoded_line = line.decode('utf-8')
+                if decoded_line.startswith('data: '):
+                    try:
+                        data = json.loads(decoded_line[6:])
+                        print(data)
+                    except json.JSONDecodeError:
+                        pass
+    ```
+  </Tabs.Tab>
+  <Tabs.Tab>
+    ```javascript
+    const apiUrl = 'http://localhost:7091/stream';
+    const apiKey = 'your_agent_api_key';
+    const question = 'your question here';
+
+    async function getStream() {
+    try {
+        const response = await fetch(apiUrl, {
+        method: 'POST',
+        headers: {
+            'Content-Type': 'application/json',
+            'Accept': 'text/event-stream'
+        },
+        body: JSON.stringify({ question, api_key: apiKey }),
+        });
+
+        if (!response.ok) {
+        throw new Error(`HTTP error! Status: ${response.status}`);
+        }
+
+        const reader = response.body.getReader();
+        const decoder = new TextDecoder();
+
+        while (true) {
+        const { done, value } = await reader.read();
+        if (done) break;
+
+        const chunk = decoder.decode(value, { stream: true });
+        // Note: This parsing method assumes each chunk contains whole lines.
+        // For a more robust production implementation, buffer the chunks
+        // and process them line by line.
+        const lines = chunk.split('\n');
+        
+        for (const line of lines) {
+            if (line.startsWith('data: ')) {
+            try {
+                const data = JSON.parse(line.substring(6));
+                console.log(data);
+            } catch (e) {
+                console.error("Failed to parse JSON from SSE event:", e);
+            }
+            }
+        }
+        }
+    } catch (error) {
+        console.error("Failed to fetch stream:", error);
+    }
+    }
+
+    getStream();
+    ```
+  </Tabs.Tab>
+</Tabs>
+
+---
+
+## Attachments API (Including Images)
+
+To attach an image (or other file) to a query:
+
+1. Upload file(s) to `/api/store_attachment` (multipart/form-data).
+2. Poll `/api/task_status` until `status=SUCCESS`.
+3. Read `result.attachment_id` from task result.
+4. Send that ID in `/stream` as `attachments: ["..."]`.
+
+<Callout type="warning">
+Attachments are processed asynchronously. Do not call `/stream` with an attachment until its task has finished with `SUCCESS`.
+</Callout>
+
+### Step 1: Upload Attachment
+
+`POST /api/store_attachment`
+
+- Content type: `multipart/form-data`
+- Form fields:
+  - `file` (required, can be repeated for multi-file upload)
+  - `api_key` (optional if JWT is present; useful for API-key-only flows)
+
+Example upload (single image):
+
+```bash
+curl -X POST http://localhost:7091/api/store_attachment \
+  -F "file=@/absolute/path/to/image.png" \
+  -F "api_key=your_agent_api_key"
+```
+
+Possible response (single-file upload):
+
+```json
+{
+  "success": true,
+  "task_id": "34f1cb56-7c7f-4d5f-a973-4ea7e65f7a10",
+  "message": "File uploaded successfully. Processing started."
+}
+```
+
+### Step 2: Poll Task Status
+
+```bash
+curl "http://localhost:7091/api/task_status?task_id=34f1cb56-7c7f-4d5f-a973-4ea7e65f7a10"
+```
+
+When complete:
+
+```json
+{
+  "status": "SUCCESS",
+  "result": {
+    "attachment_id": "67b4f8f2618dc9f19384a9e1",
+    "filename": "image.png",
+    "mime_type": "image/png"
+  }
+}
+```
+
+### Step 3: Attach to `/stream` Request
+
+Use the `attachment_id` in `attachments`.
+
+```bash
+curl -X POST http://localhost:7091/stream \
+  -H "Content-Type: application/json" \
+  -H "Accept: text/event-stream" \
+  -d '{
+    "question": "Describe this image",
+    "api_key": "your_agent_api_key",
+    "attachments": ["67b4f8f2618dc9f19384a9e1"]
+  }'
+```
+
+### Image/Attachment Behavior Notes
+
+- Typical image MIME types supported for native vision flows: `image/png`, `image/jpeg`, `image/jpg`, `image/webp`, `image/gif`.
+- If the selected model/provider does not support a file type natively, DocsGPT falls back to parsed text content.
+- For providers that support images but not native PDF file attachments, DocsGPT can convert PDF pages to images (synthetic PDF support).
+- Attachments are user-scoped. Upload and query must be done under the same user context (same API key owner or same JWT user).
+
+## Agent Portability (Export & Import)
+
+Agents can be exported to a portable YAML file and imported into another DocsGPT instance (or back into the same one). This is how you move an agent between environments or share a reproducible definition.
+
+| Method | Path | Description |
+| --- | --- | --- |
+| `GET` | `/api/export_agent?id=<agent_id>` | Download the agent as a `*.agent.yaml` file. |
+| `POST` | `/api/import_agent/plan` | Dry-run an import: parse a YAML file and return a resolution plan without creating anything. |
+| `POST` | `/api/import_agent` | Import a YAML file, creating the agent as a **draft**. |
+
+```bash
+# Export
+curl -L "https://your-docsgpt/api/export_agent?id=<agent_id>" \
+  -H "Authorization: Bearer <token>" -o my-agent.agent.yaml
+
+# Preview what an import would do
+curl -X POST https://your-docsgpt/api/import_agent/plan \
+  -H "Authorization: Bearer <token>" \
+  --data-binary @my-agent.agent.yaml
+
+# Import (creates a draft agent)
+curl -X POST https://your-docsgpt/api/import_agent \
+  -H "Authorization: Bearer <token>" \
+  --data-binary @my-agent.agent.yaml
+```
+
+Notes:
+
+- **Secrets are stripped on export** — API keys and tool credentials are never written into the YAML, so you re-enter them after import.
+- The **plan** endpoint resolves references (sources, tools, prompts) and reports what will be created or matched, so you can review before committing.
+- Imported tool URLs are validated against SSRF protections, the same as when creating tools normally.
+- **Workflow agents cannot be exported yet** — exporting one returns `400`.
+
+## Searching Conversations
+
+Search across your conversations by name and message content:
+
+```text
+GET /api/search_conversations?q=<query>&limit=30
+```
+
+| Parameter | Default | Description |
+| --- | --- | --- |
+| `q` | — (required) | Case-insensitive substring to search for. |
+| `limit` | `30` | Max results (max `100`). |
+
+Each result includes a `match_field` (`name`, `prompt`, or `response`) and a `match_snippet` showing the matched text in context, in addition to the fields returned by `/api/get_conversations`.

docs/content/Agents/basics.mdx

@@ -0,0 +1,124 @@
+---
+title: Understanding DocsGPT Agents
+description: Learn about DocsGPT Agents, their types, how to create and manage them, and how they can enhance your interaction with documents and tools.
+---
+
+import { Callout } from 'nextra/components';
+import Image from 'next/image'; // Assuming you might want to embed images later, like the ones you uploaded.
+
+# Understanding DocsGPT Agents 🤖
+
+DocsGPT Agents are advanced, configurable AI entities designed to go beyond simple question-answering. They act as specialized assistants or workers that combine instructions (prompts), knowledge (document sources), and capabilities (tools) to perform a wide range of tasks, automate workflows, and provide tailored interactions.
+
+Think of an Agent as a pre-configured version of DocsGPT, fine-tuned for a specific purpose, such as classifying documents, responding to new form submissions, or validating emails.
+
+## Why Use Agents?
+
+* **Personalization:** Create AI assistants that behave and respond according to specific roles or personas.
+* **Task Specialization:** Design agents focused on particular tasks, like customer support, data extraction, or content generation.
+* **Knowledge Integration:** Equip agents with specific document sources, making them experts in particular domains.
+* **Tool Utilization:** Grant agents access to various tools, allowing them to interact with external services, fetch live data, or perform actions.
+* **Automation:** Automate repetitive tasks by defining an agent's behavior and integrating it via webhooks or other means.
+* **Shareability:** Share your custom-configured agents with others or use agents shared with you.
+
+Agents provide a more structured and powerful way to leverage LLMs compared to a standard chat interface, as they come with a pre-defined context, instruction set, and set of capabilities.
+
+## Core Components of an Agent
+
+When you create or configure an agent, you'll work with these key components:  
+
+**Meta:**
+    * **Agent Name:** A user-friendly name to identify the agent (e.g., "Support Ticket Classifier," "Product Spec Expert").
+    * **Describe your agent:** A brief description for you or users to understand the agent's purpose.
+
+**Source:**
+    * **Select source:** The knowledge base for the agent. You can select from previously uploaded documents or data sources. This is what the agent will "know."
+    * **Chunks per query:** A numerical value determining how many relevant text chunks from the selected source are sent to the LLM with each query. This helps manage context length and relevance.
+
+**Prompt:**
+The main set of instructions or system [prompt](/Guides/Customising-prompts) that defines the agent's persona, objectives, constraints, and how it should behave or respond.
+
+**Tools:** A selection of available [DocsGPT Tools](/Tools/basics) that the agent can use to perform actions or access external information.
+
+**Agent type:** The underlying operational logic or architecture the agent uses. DocsGPT supports different types of agents, each suited for different kinds of tasks.
+
+## Understanding Agent Types
+
+DocsGPT supports several agent types, each with a distinct way of processing information. The code for these can be found in the `application/agents/` directory.
+
+### 1. Classic Agent
+
+The Classic Agent follows a traditional Retrieval Augmented Generation (RAG) approach: it retrieves relevant document chunks, augments the prompt context with them, and generates a response. It can also use configured tools if the LLM decides they are necessary.
+
+**Best for:** Direct question-answering over a specific set of documents and straightforward tool use.
+
+### 2. Agentic Agent
+
+Unlike Classic which pre-fetches documents into the prompt, the Agentic Agent gives the LLM an `internal_search` tool so it can decide **when, what, and whether** to search. This means the LLM controls its own retrieval — it can search multiple times, refine queries, or skip retrieval entirely if the question doesn't need it.
+
+**Best for:** Tasks where the agent needs to dynamically decide how to gather information, use multiple tools in sequence, or combine retrieval with external tool calls.
+
+### 3. Research Agent
+
+A multi-phase agent designed for in-depth research tasks:
+1. **Clarification** — Determines if the question needs clarification before proceeding.
+2. **Planning** — Decomposes the question into research steps with adaptive depth based on complexity.
+3. **Research** — Executes each step, calling tools and refining queries as needed.
+4. **Synthesis** — Compiles findings into a final cited report.
+
+Includes budget controls for max steps, timeout, and token limits to keep research bounded.
+
+**Best for:** Complex questions that require multi-step investigation, gathering information from multiple sources, and producing structured reports with citations.
+
+### 4. Workflow Agent
+
+Executes predefined workflows composed of connected nodes (AI Agent, Set State, Condition). See the [Workflow Nodes](/Agents/nodes) page for details on building workflows.
+
+**Best for:** Structured, multi-step processes with branching logic and shared state between steps.
+
+<Callout type="info">
+The legacy "ReAct" agent type is still accepted for backwards compatibility but maps to the Classic Agent internally. New agents should use Classic, Agentic, or Research instead.
+</Callout>
+
+## Navigating and Managing Agents in DocsGPT
+
+You can easily access and manage your agents through the DocsGPT user interface. Recently used agents appear at the top of the left sidebar for quick access. Below these, the "Manage Agents" button will take you to the main Agents page.
+
+### Creating a New Agent
+
+1.  Navigate to the "Agents" page.
+2.  Click the **"New Agent"** button.
+3.  You will be presented with the "New Agent" configuration screen:
+
+<Image
+  src="/new-agent.png"
+  alt="API Tool configuration example for phone validation"
+  width={800}
+  height={450}
+  style={{ margin: '1em auto', display: 'block', borderRadius: '8px' }}
+/>
+
+4.  Fill in the fields as described in the "Core Components of an Agent" section.
+5.  Once configured, you can **"Save Draft"** to continue editing later or **"Publish"** to make the agent active.
+
+## Interacting with and Editing Agents
+
+Once an agent is created, you can:
+
+* **Chat with it:** Select the agent to start an interaction.
+* **View Logs:** Access usage statistics, monitor token consumption per interaction, and review user message feedbacks. This is crucial for understanding how your agent is being used and performing.
+* **Edit an Agent:**
+    * Modify any of its configuration settings (name, description, source, prompt, tools, type).
+    * **Generate a Public Link:** From the edit screen, you can create a shareable public link that allows others to import and use your agent.
+    * **Get a Webhook URL:** You can also obtain a Webhook URL for the agent. This allows external applications or services to trigger the agent and receive responses programmatically, enabling powerful integrations and automations.
+* **Use it via API:** Every agent exposes an API key that can be used with the native [Agent API](/Agents/api) or the [OpenAI-Compatible API](/Agents/openai-compatible) so you can drop DocsGPT Agents into any tool that already speaks the chat completions protocol.
+
+## Seeding Premade Agents from YAML
+
+You can bootstrap a fresh DocsGPT deployment with a curated set of agents by seeding them directly into the user-data store (Postgres).
+
+1. **Customize the configuration** – edit `application/seed/config/premade_agents.yaml` (or copy from `application/seed/config/agents_template.yaml`) to describe the agents you want to provision. Each entry lets you define prompts, tools, and optional data sources.
+2. **Ensure dependencies are running** – Postgres must be reachable using `POSTGRES_URI` from `.env` (schema applied via `python scripts/db/init_postgres.py`), and a Celery worker should be available if any agent sources need to be ingested via `ingest_remote`.
+3. **Execute the seeder** – run `python -m application.seed.commands init`. Add `--force` when you need to reseed an existing environment.
+
+The seeder keeps templates under the `system` user so they appear in the UI for anyone to clone or customize. Environment variable placeholders such as `${MY_TOKEN}` inside tool configs are resolved during the seeding process.

docs/content/Agents/nodes.mdx

@@ -0,0 +1,87 @@
+# Workflow Nodes
+
+DocsGPT workflows are composed of **Nodes** that are connected to form a processing graph. These nodes interact with a **Shared State**—a global dictionary of variables that persists throughout the execution of the workflow.
+
+## The Shared State
+
+Every workflow run maintains a state object (a JSON-like dictionary).
+- **Initial State**: Contains the user's input query (`{{query}}`) and chat history (`{{chat_history}}`).
+- **Accessing Variables**: You can access any variable in the state using the double-curly braces syntax: `{{variable_name}}`.
+- **Modifying State**: Nodes read from this state and write their outputs back to it.
+
+---
+
+## AI Agent Node
+
+The **AI Agent Node** is the core processing unit. It uses a Large Language Model (LLM) to generate text, answer questions, or perform tasks using tools.
+
+### Inputs (Template Variables)
+
+The primary input is the **Prompt Template**. This field supports variable substitution.
+
+- **Prompt Template**: The text sent to the model.
+  - *Example*: `"Summarize the following text: {{user_input_text}}"`
+  - If left empty, it defaults to the initial user query (`{{query}}`).
+- **System Prompt**: Instructions that define the agent's persona and constraints.
+- **Tools**: A list of tools the agent can use (e.g., search, calculator).
+- **LLM Settings**: Specific provider, model name, and parameters.
+
+### Outputs (Emissions)
+
+When the agent completes its task, it stores the result in the shared state.
+
+- **Output Variable**: The name of the variable where the result will be saved.
+  - *Default*: If not specified, it is saved as `node_{node_id}_output`.
+  - *Custom*: You can set this to something meaningful, like `summary` or `translated_text`.
+- **Streaming**: If "Stream to user" is enabled, the output is sent to the user in real-time as it is generated, in addition to being saved to the state.
+
+### Documents
+
+An agent node can receive documents as inputs. Choose which documents the node sees:
+
+- **All**: every document attached to the run.
+- **None**: no documents.
+- **Choose**: a specific set that you select.
+
+For how a chosen document reaches the model, the node can pass it natively (send the file to a model that accepts files) or extract it to text first. The default picks automatically based on the model and the file type.
+
+---
+
+## Set State Node
+
+The **Set State Node** allows you to manipulate variables within the shared state directly without calling an LLM. This is useful for initialization, formatting, or control flow logic.
+
+### Operations
+
+You can define multiple operations in a single node. Each operation targets a specific **Key** (variable name).
+
+1.  **Set**: Assigns a specific value to a variable.
+    - *Value*: Can be a static string or a template using variables.
+    - *Example*: Set `current_step` to `1`.
+    - *Example*: Set `formatted_response` to `Analysis: {{analysis_result}}`.
+
+2.  **Increment**: Increases the value of a numeric variable.
+    - *Value*: The amount to add (default is 1).
+    - *Example*: Increment `retry_count` by `1`.
+
+3.  **Append**: Adds a value to a list variable.
+    - *Value*: The item to add to the list.
+    - *Example*: Append `{{last_result}}` to `history_list`.
+
+### Usage Examples
+
+- **Loop Counters**: Use a *Set State* node to initialize a counter (`i = 0`) before a loop, and another to increment it inside the loop.
+- **Accumulators**: Use *Append* to collect results from multiple parallel branches into a single list.
+- **Renaming**: Copy the output of a previous node to a more generic name (e.g., set `context` to `{{search_results}}`) so subsequent nodes can use a standard variable name.
+
+---
+
+## Code Node
+
+The **Code Node** runs a script in a sandboxed session bound to the workflow run. Use it to transform data, parse files, cross-check documents, or build a report that later nodes consume.
+
+- **Code**: the script to run. The workflow state is available to the script as data, so it can read variables, compute, and write results back.
+- **Inputs**: documents or artifacts to place in the workspace before the script runs. Each input accepts a produced artifact reference, a full artifact id, or an attached file.
+- **Outputs**: any files the script writes are captured as artifacts and surfaced in the run view. Write a small JSON value back to the state to pass a decision or summary to later nodes.
+
+Runs are sandboxed and have a fixed time limit, so keep each step focused. See [Artifacts and Code Execution](/Tools/artifacts-and-code-execution) for the sandbox backends and configuration.

docs/content/Agents/notifications.mdx

@@ -0,0 +1,91 @@
+---
+title: Realtime Events & Notifications (SSE)
+description: Subscribe to DocsGPT's server-sent events channel for live notifications — ingestion progress, tool approvals, MCP OAuth completion — and reconnect to an in-flight chat answer.
+---
+
+import { Callout } from 'nextra/components'
+
+# Realtime Events & Notifications
+
+DocsGPT pushes realtime updates to the browser over **Server-Sent Events (SSE)**. This is what powers the upload toasts, tool-approval prompts, and other live notifications in the UI. There are two channels:
+
+- **User events** — `GET /api/events`: a per-user notification stream (ingestion progress, tool approvals, MCP OAuth completion, …).
+- **Chat reconnect** — `GET /api/messages/<message_id>/events`: resume an answer stream that was interrupted mid-generation.
+
+<Callout type="info" emoji="ℹ️">
+Both channels require Redis (already a DocsGPT dependency). The publisher can be turned off instance-wide with `ENABLE_SSE_PUSH=false`.
+</Callout>
+
+## User events channel
+
+Open an SSE connection to receive notifications for the authenticated user:
+
+```text
+GET /api/events
+Accept: text/event-stream
+Authorization: Bearer <token>
+```
+
+Each event is a JSON object with a `type` field, for example:
+
+```text
+id: 1718900000000-0
+data: {"type":"source.ingest.queued","scope":{"id":"<source_id>"}, ...}
+```
+
+Common event types:
+
+| Type | Meaning |
+| --- | --- |
+| `source.ingest.queued` | A source ingestion task was enqueued (drives the upload toast). |
+| `mcp.oauth.completed` | An MCP server's OAuth handshake finished. |
+| tool-approval events | An agent is requesting approval to run a tool. |
+| `backlog.truncated` | The client's cursor slid off the retained backlog window — clear your cursor and refetch state. |
+
+### Reconnecting and backlog replay
+
+Events are journaled per user in a Redis Stream so a client that reconnects can catch up on what it missed. Send the last id you processed and DocsGPT replays everything after it:
+
+```text
+GET /api/events
+Last-Event-ID: 1718900000000-0
+```
+
+(You may also pass it as a `last_event_id` query parameter.) Each delivered event carries its own `id:`, so your cursor advances as you read. If you fall a long way behind, the snapshot is delivered across several reconnects rather than all at once.
+
+A few bounded behaviors to be aware of:
+
+- The backlog is capped at `EVENTS_STREAM_MAXLEN` entries (default 1000). If your `Last-Event-ID` is older than the oldest retained entry, you receive a `backlog.truncated` event — reset your cursor and refetch current state.
+- Each snapshot is capped at `EVENTS_REPLAY_MAX_PER_REQUEST` entries per request (default 200); reconnect to continue.
+- There is a per-user cap on simultaneous connections (`SSE_MAX_CONCURRENT_PER_USER`, default 8) and a windowed replay budget. Exceeding either returns **HTTP 429** — back off and retry.
+
+## Chat answer reconnect
+
+When an answer is streaming and the connection drops, resume it without losing the in-progress generation:
+
+```text
+GET /api/messages/<message_id>/events
+```
+
+This replays the message's events past your last-seen sequence number and tails the rest live. It is backed by the Postgres `message_events` journal (retained for `MESSAGE_EVENTS_RETENTION_DAYS`, default 14).
+
+<Callout type="warning" emoji="⚠️">
+The chat reconnect endpoint is a native-async route served by the ASGI entrypoint. Under a plain `flask run` dev server it returns `404`; run the backend via the ASGI app (`uvicorn application.asgi:asgi_app`) or the production gunicorn uvicorn worker to use it. See the [Development Environment](/Deploying/Development-Environment) guide.
+</Callout>
+
+## Settings
+
+| Setting | Default | Purpose |
+| --- | --- | --- |
+| `ENABLE_SSE_PUSH` | `true` | Master switch for the publisher and channel. |
+| `EVENTS_STREAM_MAXLEN` | `1000` | Per-user backlog cap (approximate). |
+| `SSE_KEEPALIVE_SECONDS` | `15` | Keepalive comment-frame cadence (keep below your proxy's idle timeout). |
+| `SSE_MAX_CONCURRENT_PER_USER` | `8` | Max simultaneous SSE connections per user (`0` disables the cap). |
+| `EVENTS_REPLAY_MAX_PER_REQUEST` | `200` | Max backlog entries per replay request. |
+| `EVENTS_REPLAY_BUDGET_REQUESTS_PER_WINDOW` | `30` | Per-user replay requests per window (`0` disables). |
+| `EVENTS_REPLAY_BUDGET_WINDOW_SECONDS` | `60` | Replay budget window length. |
+| `MESSAGE_EVENTS_RETENTION_DAYS` | `14` | Retention for the chat-stream `message_events` journal. |
+
+<Callout type="info" emoji="ℹ️">
+Operators debugging delivery issues ("the toast never appeared", "the answer didn't reconnect") can follow the SSE notifications runbook in the repository at `docs/runbooks/sse-notifications.md`.
+</Callout>

docs/content/Agents/openai-compatible.mdx

@@ -0,0 +1,239 @@
+---
+title: OpenAI-Compatible API
+description: Connect any OpenAI-compatible client to DocsGPT Agents via /v1/chat/completions — streaming, structured output, multimodal, tool calling, reasoning, and idempotent retries.
+---
+
+import { Callout, Tabs } from 'nextra/components';
+
+# OpenAI-Compatible API
+
+DocsGPT exposes `/v1/chat/completions` following the standard chat completions protocol. Point any compatible client — **opencode**, **Aider**, **LibreChat** or the OpenAI SDKs — at your DocsGPT Agent by changing only the base URL and API key.
+
+## Quick Start
+
+<Tabs items={['Python', 'cURL']}>
+  <Tabs.Tab>
+    ```python
+    from openai import OpenAI
+
+    client = OpenAI(
+        base_url="http://localhost:7091/v1",   # or https://gptcloud.arc53.com/v1
+        api_key="your_agent_api_key",
+    )
+
+    response = client.chat.completions.create(
+        model="docsgpt-agent",
+        messages=[{"role": "user", "content": "Summarize our refund policy"}],
+    )
+    print(response.choices[0].message.content)
+    ```
+  </Tabs.Tab>
+  <Tabs.Tab>
+    ```bash
+    curl -X POST http://localhost:7091/v1/chat/completions \
+      -H "Authorization: Bearer your_agent_api_key" \
+      -H "Content-Type: application/json" \
+      -d '{"model":"docsgpt-agent","messages":[{"role":"user","content":"Summarize our refund policy"}]}'
+    ```
+  </Tabs.Tab>
+</Tabs>
+
+The `model` field is accepted but ignored — the agent bound to your API key determines the model. The agent's prompt, sources, tools, and default model are loaded automatically.
+
+## Base URL & Auth
+
+| Environment | Base URL |
+| --- | --- |
+| Local | `http://localhost:7091/v1` |
+| Cloud | `https://gptcloud.arc53.com/v1` |
+
+Authenticate with `Authorization: Bearer <agent_api_key>`.
+
+## Endpoints
+
+| Method | Path | Description |
+| --- | --- | --- |
+| `POST` | `/v1/chat/completions` | Chat request (streaming or non-streaming) |
+| `GET` | `/v1/models` | List agents available to your key |
+
+## Streaming
+
+Set `"stream": true`. You'll receive SSE chunks with `choices[0].delta.content`. DocsGPT-specific events (sources, tool calls) arrive as extra frames that carry a top-level `docsgpt` key on an otherwise-empty chunk — standard clients ignore them.
+
+```python
+stream = client.chat.completions.create(
+    model="docsgpt-agent",
+    stream=True,
+    messages=[{"role": "user", "content": "Explain vector search"}],
+)
+for chunk in stream:
+    print(chunk.choices[0].delta.content or "", end="", flush=True)
+```
+
+## Sampling Parameters
+
+Standard OpenAI sampling parameters are forwarded to the model. When omitted, the agent's configured defaults apply. Supported: `temperature`, `max_tokens` (or `max_completion_tokens`), `top_p`, `frequency_penalty`, `presence_penalty`, `stop`, `seed`.
+
+```json
+{
+  "model": "docsgpt-agent",
+  "messages": [{"role": "user", "content": "Write a haiku about search"}],
+  "temperature": 0.2,
+  "max_tokens": 256,
+  "seed": 42
+}
+```
+
+## Structured Output
+
+You can force the model to return JSON matching a schema, using either the OpenAI `response_format` field or the `response_schema` convenience field.
+
+<Tabs items={['response_format', 'response_schema']}>
+  <Tabs.Tab>
+    ```json
+    {
+      "model": "docsgpt-agent",
+      "messages": [{"role": "user", "content": "Extract the order id and total"}],
+      "response_format": {
+        "type": "json_schema",
+        "json_schema": {
+          "name": "order",
+          "strict": true,
+          "schema": {
+            "type": "object",
+            "properties": {
+              "order_id": {"type": "string"},
+              "total": {"type": "number"}
+            },
+            "required": ["order_id", "total"]
+          }
+        }
+      }
+    }
+    ```
+  </Tabs.Tab>
+  <Tabs.Tab>
+    ```json
+    {
+      "model": "docsgpt-agent",
+      "messages": [{"role": "user", "content": "Extract the order id and total"}],
+      "response_schema": {
+        "type": "object",
+        "properties": {
+          "order_id": {"type": "string"},
+          "total": {"type": "number"}
+        },
+        "required": ["order_id", "total"]
+      }
+    }
+    ```
+  </Tabs.Tab>
+</Tabs>
+
+- `response_format` follows OpenAI Structured Outputs. `strict` defaults to `true`; set `strict: false` to relax enforcement.
+- `response_format: {"type": "json_object"}` requests JSON without a fixed schema (the model is steered by the prompt).
+- `response_schema` is a DocsGPT convenience: pass a raw JSON Schema object (or a `{"schema": {...}}` wrapper) directly.
+
+## Multimodal Input (text + images)
+
+User messages may use OpenAI typed-content arrays with `image_url` parts. Images are forwarded to vision-capable models.
+
+```json
+{
+  "model": "docsgpt-agent",
+  "messages": [
+    {
+      "role": "user",
+      "content": [
+        {"type": "text", "text": "What's in this screenshot?"},
+        {"type": "image_url", "image_url": {"url": "https://example.com/shot.png"}}
+      ]
+    }
+  ]
+}
+```
+
+## Tool Calling (client-side, stateless)
+
+You can register your own tools and execute them on the client. The flow is stateless — OpenAI clients that don't carry a `conversation_id` re-send the full message history each turn, and DocsGPT rebuilds the agent from it.
+
+1. Send a request with a `tools` array.
+2. If the agent decides to call a tool, the response comes back with `finish_reason: "tool_calls"` and a `tool_calls` array (and `content: null`).
+3. Execute the tool(s) on your side, then **re-POST the full message history** with the assistant's `tool_calls` message followed by `role: "tool"` result messages.
+4. DocsGPT continues the run and returns the final answer.
+
+```json
+{
+  "model": "docsgpt-agent",
+  "messages": [
+    {"role": "user", "content": "What's the weather in Paris?"},
+    {"role": "assistant", "tool_calls": [
+      {"id": "call_1", "type": "function",
+       "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}}
+    ]},
+    {"role": "tool", "tool_call_id": "call_1", "content": "18°C, clear"}
+  ],
+  "tools": [ { "type": "function", "function": { "name": "get_weather", "...": "..." } } ]
+}
+```
+
+## Reasoning
+
+For models that emit reasoning ("thinking") tokens, the response surfaces them in a non-standard `reasoning_content` field (a `reasoning_content` delta when streaming). Standard clients ignore it; clients that understand it can display the model's thinking separately from the answer.
+
+## Idempotent Retries
+
+Add an `Idempotency-Key` header so a retried request returns the *stored first response* instead of re-running the agent (which would duplicate the answer and double-bill tokens).
+
+```bash
+curl -X POST http://localhost:7091/v1/chat/completions \
+  -H "Authorization: Bearer your_agent_api_key" \
+  -H "Idempotency-Key: 8f1c...unique-per-request" \
+  -H "Content-Type: application/json" \
+  -d '{"model":"docsgpt-agent","messages":[{"role":"user","content":"hi"}]}'
+```
+
+- **Opt-in** — no header means today's behavior (every request runs).
+- **Non-streaming only** — streaming replay is not supported.
+- A completed key **replays the cached body** (and status) for **24 hours**.
+- A request with a key whose first attempt is **still in flight** returns **HTTP 409**.
+- Keys are scoped per agent and capped at **256 characters** (oversized keys are rejected).
+
+## System Prompt Override
+
+System messages are **dropped by default** — the agent's configured prompt is used. To allow callers to override it, enable **Allow prompt override** in the agent's Advanced settings.
+
+<Callout type="warning">
+When an override is active, the agent's prompt template is replaced wholesale — template variables like `{summaries}` are not substituted.
+</Callout>
+
+## Conversation Persistence
+
+Conversations are **always persisted** server-side, and the response includes `docsgpt.conversation_id`. They never appear in the agent owner's sidebar — `/v1` traffic is stored hidden, so external clients can't clutter the owner's conversation list.
+
+Stateless tool continuations (no `conversation_id`, e.g. opencode) skip persistence by default to avoid writing orphan rows; set `docsgpt.persist` to override. The legacy `docsgpt.save_conversation` flag from older releases is deprecated and ignored.
+
+## DocsGPT Extension Fields
+
+DocsGPT adds an optional `docsgpt` object to both requests and responses for features outside the OpenAI schema.
+
+**Request** (`docsgpt.*`):
+
+| Field | Description |
+| --- | --- |
+| `attachments` | List of attachment IDs to include as context for this turn. |
+| `persist` | Force-enable/disable conversation persistence (mainly for stateless tool continuations). |
+
+**Response** (`docsgpt.*`):
+
+| Field | Description |
+| --- | --- |
+| `conversation_id` | Server-side conversation ID for this exchange. |
+| `sources` | RAG sources used to answer. |
+| `tool_calls` | Completed tool-call results from the run. |
+
+When streaming, these arrive on otherwise-empty chunks that carry a top-level `docsgpt` key, so strict OpenAI clients still validate each frame.
+
+## When to Use Native Endpoints Instead
+
+Use [`/api/answer` or `/stream`](/Agents/api) if you need server-side attachments, `passthrough` template variables, explicit `conversation_id` reuse, or sidebar visibility control via `visibility`.

docs/content/Agents/webhooks.mdx

@@ -0,0 +1,174 @@
+---
+title: Triggering Agents with Webhooks
+description: Learn how to automate and integrate DocsGPT Agents using webhooks for asynchronous task execution.
+---
+
+import { Callout, Tabs } from 'nextra/components';
+
+# Triggering Agents with Webhooks
+
+Agent Webhooks provide a powerful mechanism to trigger an agent's execution from external systems. Unlike the direct API which provides an immediate response, webhooks are designed for **asynchronous** operations. When you call a webhook, DocsGPT enqueues the agent's task for background processing and immediately returns a `task_id`. You then use this ID to poll for the result.
+
+This workflow is ideal for integrating with services that expect a quick initial response (e.g., form submissions) or for triggering long-running tasks without tying up a client connection.
+
+Each agent has its own unique webhook URL, which can be generated from the agent's edit page in the DocsGPT UI. This URL includes a secure token for authentication.
+
+### API Endpoints
+
+- **Webhook URL:** `http://localhost:7091/api/webhooks/agents/{AGENT_WEBHOOK_TOKEN}`
+- **Task Status URL:** `http://localhost:7091/api/task_status`
+
+<Callout type="info">
+For DocsGPT Cloud, use `https://gptcloud.arc53.com/` as the base URL.
+</Callout>
+
+For more technical details, you can explore the API swagger documentation available for the cloud version or your local instance.
+
+---
+
+## The Webhook Workflow
+
+The process involves two main steps: triggering the task and polling for the result.
+
+### Step 1: Trigger the Webhook
+
+Send an HTTP `POST` request to the agent's unique webhook URL with the required payload. The structure of this payload should match what the agent's prompt and tools are designed to handle.
+
+-   **Method:** `POST`
+-   **Response:** A JSON object with a `task_id`. `{"task_id": "a1b2c3d4-e5f6-..."}`
+
+<Tabs items={['cURL', 'Python', 'JavaScript']}>
+  <Tabs.Tab>
+    ```bash
+    curl -X POST \
+      http://localhost:7091/api/webhooks/agents/your_webhook_token \
+      -H "Content-Type: application/json" \
+      -d '{"question": "Your message to agent"}'
+    ```
+  </Tabs.Tab>
+  <Tabs.Tab>
+    ```python
+    import requests
+
+    WEBHOOK_URL = "http://localhost:7091/api/webhooks/agents/your_webhook_token"
+    payload = {"question": "Your message to agent"}
+
+    try:
+        response = requests.post(WEBHOOK_URL, json=payload)
+        response.raise_for_status()
+        task_id = response.json().get("task_id")
+        print(f"Task successfully created with ID: {task_id}")
+    except requests.exceptions.RequestException as e:
+        print(f"Error triggering webhook: {e}")
+    ```
+  </Tabs.Tab>
+  <Tabs.Tab>
+    ```javascript
+    const webhookUrl = 'http://localhost:7091/api/webhooks/agents/your_webhook_token';
+    const payload = { question: 'Your message to agent' };
+
+    async function triggerWebhook() {
+      try {
+        const response = await fetch(webhookUrl, {
+          method: 'POST',
+          headers: { 'Content-Type': 'application/json' },
+          body: JSON.stringify(payload)
+        });
+        if (!response.ok) throw new Error(`HTTP error! ${response.status}`);
+        const data = await response.json();
+        console.log(`Task successfully created with ID: ${data.task_id}`);
+        return data.task_id;
+      } catch (error) {
+        console.error('Error triggering webhook:', error);
+      }
+    }
+    
+    triggerWebhook();
+    ```
+  </Tabs.Tab>
+</Tabs>
+
+### Triggering with GET (query parameters)
+
+The webhook listener also accepts `GET` requests, mapping the query string to the payload. This is handy for systems that can only issue a simple `GET` (for example a "ping this URL" integration):
+
+```bash
+curl "http://localhost:7091/api/webhooks/agents/your_webhook_token?question=Your+message+to+agent"
+```
+
+As with `POST`, the response is a JSON object containing the `task_id`.
+
+### Preventing duplicate triggers (Idempotency-Key)
+
+To make a trigger safe to retry, send an optional `Idempotency-Key` header. If the same key is seen again within 24 hours, DocsGPT does not enqueue a second task — it returns the **original** `task_id` instead. This prevents a retried or double-fired webhook from running the agent twice.
+
+```bash
+curl -X POST \
+  http://localhost:7091/api/webhooks/agents/your_webhook_token \
+  -H "Content-Type: application/json" \
+  -H "Idempotency-Key: order-4567-created" \
+  -d '{"question": "Your message to agent"}'
+```
+
+### Step 2: Poll for the Result
+
+Once you have the `task_id`, periodically send a `GET` request to the `/api/task_status` endpoint until the task `status` is `SUCCESS` or `FAILURE`.
+
+- **`status`**: The current state of the task (`PENDING`, `STARTED`, `SUCCESS`, `FAILURE`).
+- **`result`**: The final output from the agent, available when the status is `SUCCESS` or `FAILURE`.
+
+<Tabs items={['cURL', 'Python', 'JavaScript']}>
+  <Tabs.Tab>
+    ```bash
+    # Replace the task_id with the one you received
+    curl http://localhost:7091/api/task_status?task_id=YOUR_TASK_ID
+    ```
+  </Tabs.Tab>
+  <Tabs.Tab>
+    ```python
+    import requests
+    import time
+
+    STATUS_URL = "http://localhost:7091/api/task_status"
+    task_id = "YOUR_TASK_ID"
+
+    while True:
+        response = requests.get(STATUS_URL, params={"task_id": task_id})
+        data = response.json()
+        status = data.get("status")
+        print(f"Current task status: {status}")
+
+        if status in ["SUCCESS", "FAILURE"]:
+            print("Final Result:")
+            print(data.get("result"))
+            break
+        
+        time.sleep(2)
+    ```
+  </Tabs.Tab>
+  <Tabs.Tab>
+    ```javascript
+    const statusUrl = 'http://localhost:7091/api/task_status';
+    const taskId = 'YOUR_TASK_ID';
+
+    const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));
+
+    async function pollForResult() {
+      while (true) {
+        const response = await fetch(`${statusUrl}?task_id=${taskId}`);
+        const data = await response.json();
+        const status = data.status;
+        console.log(`Current task status: ${status}`);
+
+        if (status === 'SUCCESS' || status === 'FAILURE') {
+          console.log('Final Result:', data.result);
+          break;
+        }
+        await sleep(2000);
+      }
+    }
+
+    pollForResult();
+    ```
+  </Tabs.Tab>
+</Tabs>

docs/content/Deploying/Access-Control.mdx

@@ -0,0 +1,149 @@
+---
+title: Access Control, Roles & Teams
+description: How DocsGPT models permissions — global admin/user roles (RBAC), the admin dashboard, teams with per-team roles, and sharing agents, sources, prompts, and tools.
+---
+
+import { Callout } from 'nextra/components'
+
+# Access Control, Roles & Teams
+
+DocsGPT has two independent permission planes:
+
+- **Global RBAC** — every user holds the `admin` or `user` role for the whole instance. Admins can manage other users and see instance-wide usage and audit data.
+- **Team roles** — within a team a user is a `team_admin` or `team_member`, and resources can be shared to teams or individual members.
+
+The two planes never mix: a global `admin` is a superuser over *all* teams, but a `team_admin` is **not** a global admin.
+
+<Callout type="info" emoji="🔒">
+Roles are resolved **server-side on every request** and are never trusted from the JWT. The frontend route guards are cosmetic — the server-side decorators are the real boundary. Persisted roles apply under `AUTH_TYPE=oidc`; token-only modes (`simple_jwt`, `session_jwt`) can never hold the admin role.
+</Callout>
+
+## Global roles (RBAC)
+
+| Role | Capabilities |
+| --- | --- |
+| `user` | The default. Owns their own conversations, sources, agents, prompts, and tools. |
+| `admin` | Everything a user can do, plus the [admin dashboard](#admin-dashboard): user management, force-logout, role management, and instance-wide usage/audit. |
+
+To see the roles the current request resolves to, call:
+
+```text
+GET /api/user/me   →   { "user_id": "...", "roles": ["user"], "email": "...", "name": "...", "picture": "..." }
+```
+
+This is the canonical way to check a caller's effective roles (distinct from `/api/config`, which only reports the instance `auth_type`).
+
+## Granting admin
+
+There are four ways an account becomes an admin:
+
+1. **The `grant_admin` script** — run `python scripts/grant_admin.py <user_id>` on the server to grant (or `--revoke` / `--list`) the admin role directly. This is the canonical bootstrap for the first admin.
+2. **OIDC group mapping** — `OIDC_ADMIN_GROUPS` auto-grants admin to members of the listed IdP groups. See [below](#admin-via-oidc-groups).
+3. **Local no-auth mode** — `LOCAL_MODE_ADMIN=true` grants admin when `AUTH_TYPE=None` (self-host, no authentication).
+4. **Grant by an existing admin** — through the admin API/dashboard.
+
+| Setting | Default | Description |
+| --- | --- | --- |
+| `OIDC_ADMIN_GROUPS` | — | Comma-separated IdP groups whose members are granted the global `admin` role. Re-checked at every login and silent renewal. Unset = no OIDC admin mapping. |
+| `LOCAL_MODE_ADMIN` | `false` | Grants admin in no-auth mode only (`AUTH_TYPE=None`). |
+
+<Callout type="warning" emoji="⚠️">
+**Never set `LOCAL_MODE_ADMIN=true` on a networked deployment.** It only makes sense for a single-user local install with `AUTH_TYPE=None`, where there is no identity to check.
+</Callout>
+
+### Bootstrapping the first admin
+
+Granting admin through the dashboard itself requires *already being* an admin, which creates a chicken-and-egg problem on a fresh deployment. Break it one of these ways:
+
+- **Run the script** on the server: `python scripts/grant_admin.py <user_id>` (the `user_id` is the OIDC `sub`). The grant is written to `user_roles` with `source='manual'` and takes effect on the user's next request. Use `--list` to see current admins and `--revoke` to remove a manual grant.
+- Set `OIDC_ADMIN_GROUPS` to a group you belong to, and sign in — you become an admin automatically.
+- For a no-auth local install, use `LOCAL_MODE_ADMIN=true`.
+
+### Admin via OIDC groups
+
+When `OIDC_ADMIN_GROUPS` is set, group membership is mapped to the admin role at every sign-in *and* every [silent renewal](/Deploying/OIDC-SSO#silent-session-renewal) — so removing a user from the admin group revokes their admin at the next renewal, just like the [sign-in allowlist](/Deploying/OIDC-SSO#restricting-sign-in-by-group). It is independent of `OIDC_ALLOWED_GROUPS` (which controls *whether* a user may sign in at all). Leaving `OIDC_ADMIN_GROUPS` unset never mass-revokes admin.
+
+```env
+OIDC_ADMIN_GROUPS=platform-admins
+# OIDC_GROUPS_CLAIM=groups   # only if your IdP uses a different claim name
+```
+
+If your IdP only exposes groups via the userinfo endpoint, DocsGPT backfills them from there during reconciliation, the same as the allowlist.
+
+## Admin dashboard
+
+Admins get a dashboard backed by a REST surface under `/api/admin` (every endpoint requires the admin role). All mutating actions are written to the `auth_events` audit log with the acting admin recorded in the metadata.
+
+| Method | Path | Description |
+| --- | --- | --- |
+| `GET` | `/api/admin/overview` | Instance overview counts. |
+| `GET` | `/api/admin/users` | List users (paginated; supports a `user_id` filter). |
+| `GET` | `/api/admin/users/<id>` | Drill into a single user. |
+| `PATCH` | `/api/admin/users/<id>` | Activate / deactivate a user (deactivation revokes their sessions). |
+| `POST` | `/api/admin/users/<id>/role` | Grant the admin role (guards against removing the last admin). |
+| `DELETE` | `/api/admin/users/<id>/role` | Revoke the admin role. |
+| `POST` | `/api/admin/users/<id>/revoke-sessions` | Force-logout a user. |
+| `GET` | `/api/admin/admins` | List current admins. |
+| `GET` | `/api/admin/usage` | Token-usage series and top users. |
+| `GET` | `/api/admin/audit` | Authentication/admin audit feed. |
+| `GET` | `/api/admin/devices/audit` | Remote-device audit feed. |
+| `GET` | `/api/admin/teams` | Instance-wide oversight of all teams. |
+
+<Callout type="info" emoji="ℹ️">
+Deactivating a user via the dashboard works for any auth type, while OIDC deployments can also offboard through [SCIM](/Deploying/OIDC-SSO#scim-user-provisioning). Both revoke live sessions immediately.
+</Callout>
+
+## Teams
+
+Teams let a group of users collaborate and share resources. Teams are self-serve — any user can create one and manage its membership.
+
+### Team roles
+
+| Role | Capabilities |
+| --- | --- |
+| `team_member` | Belongs to the team; can use resources shared with the team. |
+| `team_admin` | Manages membership and team settings. Implies `team_member`. |
+
+The team **owner** is a distinct concept from `team_admin`: ownership can be transferred, and a global `admin` is treated as a superuser over every team.
+
+### Managing a team
+
+| Method | Path | Description |
+| --- | --- | --- |
+| `GET` / `POST` | `/api/teams` | List your teams / create a team. |
+| `GET` / `PATCH` / `DELETE` | `/api/teams/<id>` | Read, update, or delete a team. |
+| `GET` / `POST` | `/api/teams/<id>/members` | List members / add a member. |
+| `PATCH` / `DELETE` | `/api/teams/<id>/members/<member_id>` | Change a member's role / remove (or leave). |
+| `POST` | `/api/teams/<id>/transfer_owner` | Transfer ownership. |
+
+Membership and removal are guarded so a team can never be left without an admin (the last-admin guard).
+
+<Callout type="info" emoji="ℹ️">
+Members are added by email (or subject id). The invitee **must have signed in at least once** so DocsGPT can resolve them to a user account.
+</Callout>
+
+## Sharing resources with a team
+
+Four resource types can be shared: **agents**, **sources**, **prompts**, and **tools**. Sharing is **additive** — it grants access to others without changing ownership.
+
+| Method | Path | Description |
+| --- | --- | --- |
+| `GET` / `POST` / `DELETE` | `/api/teams/<id>/grants` | List, create, or revoke a share within a team. |
+| `GET` | `/api/resource_shares` | List shares visible to the caller. |
+
+Sharing rules:
+
+- Only the **owner** of a resource can share it.
+- A share targets either the **whole team** or a **single member**.
+- Each share carries an access level: **`viewer`** (read-only) or **`editor`** (read and modify).
+- `editor` is not the same as owner — an editor can change a resource but cannot delete it or re-share it.
+- Shared tools run server-side with the **owner's** credentials; a grantee never sees the owner's secrets.
+
+## Audit log
+
+Access-control actions are appended to the `auth_events` table alongside the [authentication events](/Deploying/OIDC-SSO#login-auditing). This includes admin actions — `admin_user_activated` / `admin_user_deactivated`, `admin_sessions_revoked`, `role_granted` / `role_revoked` (with `metadata.source` = `manual` or `oidc_group`) — and team events (`team.create`, `team.member_add`, `team.member_role`, `team.member_remove`, `team.share`, `team.unshare`, `team.transfer_owner`, `team.delete`). The acting admin is recorded in the event metadata.
+
+## Related
+
+- [SSO with OIDC](/Deploying/OIDC-SSO) — sign-in, group allowlists, and the `auth_events` table.
+- [App Configuration](/Deploying/DocsGPT-Settings) — the full settings reference.

docs/content/Deploying/Amazon-Lightsail.mdx

@@ -0,0 +1,110 @@
+---
+title: Hosting DocsGPT on Amazon Lightsail
+description:
+display: hidden
+---
+
+# Self-hosting DocsGPT on Amazon Lightsail
+
+Here's a step-by-step guide on how to set up an Amazon Lightsail instance to host DocsGPT.
+
+## Configuring your instance
+
+(If you know how to create a Lightsail instance, you can skip to the recommended configuration part by clicking [here](#connecting-to-your-newly-created-instance)).
+
+### 1. Create an AWS Account: 
+If you haven't already, create or log in to your AWS account at https://lightsail.aws.amazon.com.
+
+### 2. Create an Instance: 
+
+a. Click "Create Instance."
+
+b. Select the "Instance location." In most cases, the default location works fine.
+
+c. Choose "Linux/Unix" as the image and "Ubuntu 20.04 LTS" as the Operating System.
+
+d. Configure the instance plan based on your requirements. A "1 GB, 1vCPU, 40GB SSD, and 2TB transfer" setup is recommended for most scenarios.
+
+e. Give your instance a unique name and click "Create Instance."
+
+PS: It may take a few minutes for the instance setup to complete.
+
+### Connecting to Your newly created Instance
+
+Your instance will be ready a few minutes after creation. To access it, open the instance and click "Connect using SSH."
+
+#### Clone the DocsGPT Repository
+
+A terminal window will pop up, and the first step will be to clone the DocsGPT Git repository:
+
+`git clone https://github.com/arc53/DocsGPT.git`
+
+#### Download the package information
+
+Once it has finished cloning the repository, it is time to download the package information from all sources. To do so, simply enter the following command:
+
+`sudo apt update`
+
+#### Install Docker and Docker Compose
+
+DocsGPT backend and worker use Python, Frontend is written on React and the whole application is containerized using Docker. To install Docker and Docker Compose, enter the following commands:
+
+`sudo apt install docker.io`
+
+And now install docker-compose:
+
+`sudo apt install docker-compose`
+
+#### Access the DocsGPT Folder
+
+Enter the following command to access the folder in which the DocsGPT docker-compose file is present.
+
+`cd DocsGPT/`
+
+#### Prepare the Environment
+
+Inside the DocsGPT folder create a `.env` file and copy the contents of `.env_sample` into it.
+
+`nano .env`
+
+Make sure your `.env` file looks like this:
+
+```
+API_KEY=<Your LLM API key>
+LLM_NAME=docsgpt
+VITE_API_STREAMING=true
+```
+
+To save the file, press CTRL+X, then Y, and then ENTER.
+
+Next, set the correct IP for the Backend by opening the docker-compose.yml file:
+
+`nano deployment/docker-compose.yaml`
+
+And Change line 7 to: `VITE_API_HOST=http://localhost:7091`
+to this `VITE_API_HOST=http://<your instance public IP>:7091`
+
+This will allow the frontend to connect to the backend.
+
+#### Running the Application
+
+You're almost there! Now that all the necessary bits and pieces have been installed, it is time to run the application. To do so, use the following command:
+
+`sudo docker compose -f deployment/docker-compose.yaml up -d`
+
+Launching it for the first time will take a few minutes to download all the necessary dependencies and build.
+
+Once this is done you can go ahead and close the terminal window.
+
+#### Enabling Ports 
+
+a. Before you are able to access your live instance, you must first enable the port that it is using.
+
+b. Open your Lightsail instance and head to "Networking".
+
+c. Then click on "Add rule" under "IPv4 Firewall", enter `5173` as your port, and hit "Create". 
+Repeat the process for port `7091`.
+
+#### Access your instance
+
+Your instance is now available at your Public IP Address on port 5173. Enjoy using DocsGPT!

docs/content/Deploying/Development-Environment.mdx

@@ -0,0 +1,188 @@
+---
+title: Setting Up a Development Environment
+description: Guide to setting up a development environment for DocsGPT, including backend and frontend setup.
+---
+
+import { Callout } from 'nextra/components'
+
+# Setting Up a Development Environment
+
+This guide will walk you through setting up a development environment for DocsGPT. This setup allows you to modify and test the application's backend and frontend components.
+
+## 1. Spin Up Postgres and Redis
+
+For development purposes, you can quickly start Postgres and Redis containers. Postgres is the user-data store for DocsGPT (conversations, agents, prompts, sources, attachments, workflows, logs, and token usage), and Redis is used as the cache and Celery broker. We provide a dedicated Docker Compose file, `docker-compose-dev.yaml`, located in the `deployment` directory, that includes only these essential services. The backend applies the Alembic schema automatically on first boot (`AUTO_MIGRATE=true` / `AUTO_CREATE_DB=true` ship enabled), so no separate migration step is required. You can still run `python scripts/db/init_postgres.py` explicitly if you prefer.
+
+You can find the `docker-compose-dev.yaml` file [here](https://github.com/arc53/DocsGPT/blob/main/deployment/docker-compose-dev.yaml).
+
+**Steps to start Postgres and Redis:**
+
+1.  Navigate to the root directory of your DocsGPT repository in your terminal.
+
+2.  Run the following commands to build and start the containers defined in `docker-compose-dev.yaml`:
+
+    ```bash
+    docker compose -f deployment/docker-compose-dev.yaml build
+    docker compose -f deployment/docker-compose-dev.yaml up -d
+    ```
+
+    These commands will start Postgres and Redis in detached mode, running in the background. When the Flask backend boots against the fresh Postgres instance, it will automatically create the database (if missing) and apply the current Alembic schema.
+
+<Callout type="info" emoji="ℹ️">
+  MongoDB is no longer required for a default DocsGPT install. If you
+  specifically want to use MongoDB Atlas as the vector store
+  (`VECTOR_STORE=mongodb`), start it on the side via
+  `deployment/docker-compose.mongo.yaml`. For migrating an existing
+  Mongo-based install to Postgres, see
+  [PostgreSQL for User Data](/Deploying/Postgres-Migration).
+</Callout>
+
+## 2. Run the Backend
+
+To run the DocsGPT backend locally, you'll need to set up a Python environment and install the necessary dependencies.
+
+**Prerequisites:**
+
+*   **Python 3.12:** Ensure you have Python 3.12 installed on your system. You can check your Python version by running `python --version` or `python3 --version` in your terminal.
+
+**Steps to run the backend:**
+
+1.  **Configure Environment Variables:**
+
+    DocsGPT backend settings are configured using environment variables. You can set these either in a `.env` file or directly in the `settings.py` file. For a comprehensive overview of all settings, please refer to the [DocsGPT Settings Guide](/Deploying/DocsGPT-Settings).
+
+    *   **Option 1: Using a `.env` file (Recommended):**
+        *   If you haven't already, create a file named `.env` in the **root directory** of your DocsGPT project.
+        *   Modify the `.env` file to adjust settings as needed. You can find a comprehensive list of configurable options in [`application/core/settings.py`](https://github.com/arc53/DocsGPT/blob/main/application/core/settings.py).
+
+    *   **Option 2: Exporting Environment Variables:**
+        *   Alternatively, you can export environment variables directly in your terminal. However, using a `.env` file is generally more organized for development.
+
+2.  **Create a Python Virtual Environment (Optional but Recommended):**
+
+    Using a virtual environment isolates project dependencies and avoids conflicts with system-wide Python packages.
+
+    *   **macOS and Linux:**
+
+        ```bash
+        python -m venv venv
+        . venv/bin/activate
+        ```
+
+    *   **Windows:**
+
+        ```bash
+        python -m venv venv
+         venv/Scripts/activate
+        ```
+
+3.  **Download Embedding Model:**
+
+    The backend requires an embedding model. Download the `mpnet-base-v2` model and place it in the `models/` directory within the project root. You can use the following script:
+
+    ```bash
+    wget https://d3dg1063dc54p9.cloudfront.net/models/embeddings/mpnet-base-v2.zip
+    unzip mpnet-base-v2.zip -d model
+    rm mpnet-base-v2.zip
+    ```
+
+    Alternatively, you can manually download the zip file from [here](https://d3dg1063dc54p9.cloudfront.net/models/embeddings/mpnet-base-v2.zip), unzip it, and place the extracted folder in `models/`.
+
+4.  **Install Backend Dependencies:**
+
+    Navigate to the root of your DocsGPT repository and install the required Python packages:
+
+    ```bash
+    pip install -r application/requirements.txt
+    ```
+
+5.  **Run the Backend:**
+
+    For local development, run the ASGI composition under uvicorn. It serves the **whole** application, hot-reloads on source changes, and matches the production runtime:
+
+    ```bash
+    uvicorn application.asgi:asgi_app --host 0.0.0.0 --port 7091 --reload
+    ```
+
+    This makes the backend accessible on `http://localhost:7091`. Production uses `gunicorn -k uvicorn_worker.UvicornWorker` against the same `application.asgi:asgi_app` target.
+
+    A plain Flask run is a faster inner loop (quick startup, the Werkzeug interactive debugger):
+
+    ```bash
+    flask --app application/app.py run --host=0.0.0.0 --port=7091
+    ```
+
+    But it serves **only** the WSGI Flask app and omits the routes mounted on the ASGI shell in `application/asgi.py`: the `/mcp` FastMCP endpoint and the native-async SSE reconnect reader `GET /api/messages/<id>/events`. Under `flask run` those paths return 404 — chat still works (`POST /stream` is a Flask route), but a stream interrupted by a disconnect won't auto-resume on reconnect. Use `flask run` only when you don't need those routes.
+
+6.  **Start the Celery Worker:**
+
+    Open a new terminal window (and activate your virtual environment if you used one). Start the Celery worker to handle background tasks:
+
+    ```bash
+    celery -A application.app.celery worker -l INFO
+    ```
+
+    This command will start the Celery worker, which processes tasks such as document parsing and vector embedding.
+
+    **macOS note:** Due to a threading issue, start Celery with the solo pool:
+
+    ```bash
+    python -m celery -A application.app.celery worker -l INFO --pool=solo
+    ```
+
+**Running in Debugger (VSCode):**
+
+For easier debugging, you can launch the Flask app and Celery worker directly from VSCode's debugger.
+
+*   Press <kbd>Shift</kbd> + <kbd>Cmd</kbd> + <kbd>D</kbd> (macOS) or <kbd>Shift</kbd> + <kbd>Windows</kbd> + <kbd>D</kbd> (Windows) to open the Run and Debug view.
+*   You should see configurations named "Flask" and "Celery". Select the desired configuration and click the "Start Debugging" button (green play icon).
+
+## 3. Start the Frontend
+
+To run the DocsGPT frontend locally, you'll need Node.js and npm (Node Package Manager).
+
+**Prerequisites:**
+
+*   **Node.js version 16 or higher:** Ensure you have Node.js version 16 or greater installed. You can check your Node.js version by running `node -v` in your terminal. npm is usually bundled with Node.js.
+
+**Steps to start the frontend:**
+
+1.  **Navigate to the Frontend Directory:**
+
+    In your terminal, change the current directory to the `frontend` folder within your DocsGPT repository:
+
+    ```bash
+    cd frontend
+    ```
+
+2.  **Install Global Packages (If Needed):**
+
+    If you don't have `husky` and `vite` installed globally, you can install them:
+
+    ```bash
+    npm install husky -g
+    npm install vite -g
+    ```
+    You can skip this step if you already have these packages installed or prefer to use local installations (though global installation simplifies running the commands in this guide).
+
+3.  **Install Frontend Dependencies:**
+
+    Install the project's frontend dependencies using npm:
+
+    ```bash
+    npm install --include=dev
+    ```
+
+    This command reads the `package.json` file in the `frontend` directory and installs all listed dependencies, including development dependencies.
+
+4.  **Run the Frontend App:**
+
+    Start the frontend development server:
+
+    ```bash
+    npm run dev
+    ```
+
+    This command will start the Vite development server. The frontend application will typically be accessible at [http://localhost:5173/](http://localhost:5173/). The terminal will display the exact URL where the frontend is running.
+
+With both the backend and frontend running, you should now have a fully functional DocsGPT development environment. You can access the application in your browser at [http://localhost:5173/](http://localhost:5173/) and start developing!

docs/content/Deploying/Docker-Deploying.mdx

@@ -0,0 +1,135 @@
+---
+title: Docker Deployment of DocsGPT
+description: Deploy DocsGPT using Docker and Docker Compose for easy setup and management.
+---
+
+# Docker Deployment of DocsGPT
+
+Docker is the recommended method for deploying DocsGPT, providing a consistent and isolated environment for the application to run. This guide will walk you through deploying DocsGPT using Docker and Docker Compose.
+
+## Prerequisites
+
+* **Docker Engine:** You need to have Docker Engine installed on your system.
+    * **macOS:** [Docker Desktop for Mac](https://docs.docker.com/desktop/install/mac-install/)
+    * **Linux:** [Docker Engine Installation Guide](https://docs.docker.com/engine/install/) (follow instructions for your specific distribution)
+    * **Windows:** [Docker Desktop for Windows](https://docs.docker.com/desktop/install/windows-install/) (requires WSL 2 backend, see notes below)
+* **Docker Compose:** Docker Compose is usually included with Docker Desktop. If you are using Docker Engine separately, ensure you have Docker Compose V2 installed.
+
+**Important Note for Windows Users:** Docker Desktop on Windows generally requires the WSL 2 backend to function correctly, especially when using features like host networking which are utilized in DocsGPT's Docker Compose setup. Ensure WSL 2 is enabled and configured in Docker Desktop settings.
+
+## Quickest Setup: Using DocsGPT Public API
+
+The fastest way to try out DocsGPT is by using the public API endpoint. This requires minimal configuration and no local LLM setup.
+
+1.  **Clone the DocsGPT Repository (if you haven't already):**
+
+    ```bash
+    git clone https://github.com/arc53/DocsGPT.git
+    cd DocsGPT
+    ```
+
+2.  **Create a `.env` file:**
+
+    In the root directory of your DocsGPT repository, create a file named `.env`.
+
+3.  **Add Public API Configuration to `.env`:**
+
+    Open the `.env` file and add the following lines:
+
+    ```
+    LLM_PROVIDER=docsgpt
+    VITE_API_STREAMING=true
+    ```
+
+    This minimal configuration tells DocsGPT to use the public API. For more advanced settings and other LLM options, refer to the [DocsGPT Settings Guide](/Deploying/DocsGPT-Settings).
+
+4.  **Launch DocsGPT with Docker Compose:**
+
+    Navigate to the root directory of the DocsGPT repository in your terminal and run:
+
+    ```bash
+    docker compose -f deployment/docker-compose.yaml up -d
+    ```
+
+    The `-d` flag runs Docker Compose in detached mode (in the background).
+
+5.  **Access DocsGPT in your browser:**
+
+    Once the containers are running, open your web browser and go to [http://localhost:5173/](http://localhost:5173/).
+
+6.  **Stopping DocsGPT:**
+
+    To stop the application, navigate to the same directory in your terminal and run:
+
+    ```bash
+    docker compose -f deployment/docker-compose.yaml down
+    ```
+
+## Optional Ollama Setup (Local Models)
+
+DocsGPT provides optional Docker Compose files to easily integrate with [Ollama](https://ollama.com/) for running local models. These files add an official Ollama container to your Docker Compose setup. These files are located in the `deployment/optional/` directory.
+
+There are two Ollama optional files:
+
+*   **`docker-compose.optional.ollama-cpu.yaml`**: For running Ollama on CPU.
+*   **`docker-compose.optional.ollama-gpu.yaml`**: For running Ollama on GPU (requires Docker to be configured for GPU usage).
+
+### Launching with Ollama and Pulling a Model
+
+1.  **Clone the DocsGPT Repository and Create `.env` (as described above).**
+
+2.  **Launch DocsGPT with Ollama Docker Compose:**
+
+    Choose the appropriate Ollama Compose file (CPU or GPU) and launch DocsGPT:
+
+    **CPU:**
+    ```bash
+    docker compose --env-file .env -f deployment/docker-compose.yaml -f deployment/optional/docker-compose.optional.ollama-cpu.yaml up -d
+    ```
+    **GPU:**
+    ```bash
+    docker compose --env-file .env -f deployment/docker-compose.yaml -f deployment/optional/docker-compose.optional.ollama-gpu.yaml up -d
+    ```
+
+3.  **Pull the Ollama Model:**
+
+    **Crucially, after launching with Ollama, you need to pull the desired model into the Ollama container.**  Find the `LLM_NAME` you configured in your `.env` file (e.g., `llama3.2:1b`). Then execute the following command to pull the model *inside* the running Ollama container:
+
+    ```bash
+    docker compose -f deployment/docker-compose.yaml -f deployment/optional/docker-compose.optional.ollama-cpu.yaml exec -it ollama ollama pull <LLM_NAME>
+    ```
+    or (for GPU):
+     ```bash
+    docker compose -f deployment/docker-compose.yaml -f deployment/optional/docker-compose.optional.ollama-gpu.yaml exec -it ollama ollama pull <LLM_NAME>
+    ```
+    Replace `<LLM_NAME>` with the actual model name from your `.env` file.
+
+4.  **Access DocsGPT in your browser:**
+
+    Once the model is pulled and containers are running, open your web browser and go to [http://localhost:5173/](http://localhost:5173/).
+
+5.  **Stopping Ollama Setup:**
+
+    To stop a DocsGPT setup launched with Ollama optional files, use `docker compose down` and include all the compose files used during the `up` command:
+
+    ```bash
+    docker compose -f deployment/docker-compose.yaml -f deployment/optional/docker-compose.optional.ollama-cpu.yaml down
+    ```
+    or
+
+    ```bash
+    docker compose -f deployment/docker-compose.yaml -f deployment/optional/docker-compose.optional.ollama-gpu.yaml down
+    ```
+
+**Important for GPU Usage:**
+
+*   **NVIDIA Container Toolkit (for NVIDIA GPUs):** If you are using NVIDIA GPUs, you need to have the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) installed and configured on your system for Docker to access your GPU.
+*   **Docker GPU Configuration:** Ensure Docker is configured to utilize your GPU. Refer to the [Ollama Docker Hub page](https://hub.docker.com/r/ollama/ollama) and Docker documentation for GPU setup instructions specific to your GPU type (NVIDIA, AMD, Intel).
+
+## Restarting After Configuration Changes
+
+Whenever you modify the `.env` file or any Docker Compose files, you need to restart the Docker containers for the changes to be applied.  Use the same `docker compose down` and `docker compose up -d` commands you used to launch DocsGPT, ensuring you include all relevant `-f` flags for optional files if you are using them.
+
+## Further Configuration
+
+This guide covers the basic Docker deployment of DocsGPT. For detailed information on configuring various aspects of DocsGPT, such as LLM providers, models, vector stores, and more, please refer to the comprehensive [DocsGPT Settings Guide](/Deploying/DocsGPT-Settings).

docs/content/Deploying/DocsGPT-Settings.mdx

@@ -0,0 +1,448 @@
+---
+title: DocsGPT Settings
+description: Configure your DocsGPT application by understanding the basic settings.
+---
+
+import { Callout } from 'nextra/components'
+
+# DocsGPT Settings
+
+DocsGPT is highly configurable, allowing you to tailor it to your specific needs and preferences. You can control various aspects of the application, from choosing the Large Language Model (LLM) provider to selecting embedding models and vector stores.
+
+This document will guide you through the basic settings you can configure in DocsGPT. These settings determine how DocsGPT interacts with LLMs and processes your data.
+
+## Configuration Methods
+
+There are two primary ways to configure DocsGPT settings:
+
+### 1. Configuration via `.env` file (Recommended)
+
+The easiest and recommended way to configure basic settings is by using a `.env` file. This file should be located in the **root directory** of your DocsGPT project (the same directory where `setup.sh` is located).
+
+**Example `.env` file structure:**
+
+```
+LLM_PROVIDER=openai
+API_KEY=YOUR_OPENAI_API_KEY
+LLM_NAME=gpt-4o
+```
+
+### 2. Configuration via `settings.py` file (Advanced)
+
+For more advanced configurations or if you prefer to manage settings directly in code, you can modify the `settings.py` file. This file is located in the `application/core` directory of your DocsGPT project.
+
+While modifying `settings.py` offers more flexibility, it's generally recommended to use the `.env` file for basic settings and reserve `settings.py` for more complex adjustments or when you need to configure settings programmatically.
+
+**Location of `settings.py`:** `application/core/settings.py`
+
+## Basic Settings Explained
+
+Here are some of the most fundamental settings you'll likely want to configure:
+
+- **`LLM_PROVIDER`**: This setting determines which Large Language Model (LLM) provider DocsGPT will use. It tells DocsGPT which API to interact with.
+
+  - **Common values:**
+    - `docsgpt`: Use the DocsGPT Public API Endpoint (simple and free, as offered in `setup.sh` option 1).
+    - `openai`: Use OpenAI's API (requires an API key).
+    - `google`: Use Google's Vertex AI or Gemini models.
+    - `anthropic`: Use Anthropic's Claude models.
+    - `groq`: Use Groq's models.
+    - `huggingface`: Use HuggingFace Inference API.
+    - `openai` (when using local inference engines like Ollama, Llama.cpp, TGI, etc.): This signals DocsGPT to use an OpenAI-compatible API format, even if the actual LLM is running locally.
+
+- **`LLM_NAME`**: Specifies the specific model to use from the chosen LLM provider. The available models depend on the `LLM_PROVIDER` you've selected.
+
+  - **Examples:**
+    - For `LLM_PROVIDER=openai`: `gpt-4o`
+    - For `LLM_PROVIDER=google`: `gemini-3.5-flash`
+    - For local models (e.g., Ollama): `llama3.2:1b` (or any model name available in your setup).
+
+- **`EMBEDDINGS_NAME`**: This setting defines which embedding model DocsGPT will use to generate vector embeddings for your documents. Embeddings are numerical representations of text that allow DocsGPT to understand the semantic meaning of your documents for efficient search and retrieval.
+
+  - **Default value:** `huggingface_sentence-transformers/all-mpnet-base-v2` (a good general-purpose embedding model).
+  - **Other options:** You can explore other embedding models from Hugging Face Sentence Transformers or other providers if needed.
+
+- **`API_KEY`**: Required for most cloud-based LLM providers. This is your authentication key to access the LLM provider's API. You'll need to obtain this key from your chosen provider's platform.
+
+- **`OPENAI_BASE_URL`**: Specifically used when `LLM_PROVIDER` is set to `openai` but you are connecting to a local inference engine (like Ollama, Llama.cpp, etc.) that exposes an OpenAI-compatible API. This setting tells DocsGPT where to find your local LLM server.
+
+- **`STT_PROVIDER`**: Selects the speech-to-text provider used for microphone transcription in chat and for audio file ingestion through the parser pipeline.
+
+## Configuration Examples
+
+Let's look at some concrete examples of how to configure these settings in your `.env` file.
+
+### Example for Cloud API Provider (OpenAI)
+
+To use OpenAI's `gpt-4o` model, you would configure your `.env` file like this:
+
+```
+LLM_PROVIDER=openai
+API_KEY=YOUR_OPENAI_API_KEY  # Replace with your actual OpenAI API key
+LLM_NAME=gpt-4o
+```
+
+Make sure to replace `YOUR_OPENAI_API_KEY` with your actual OpenAI API key.
+
+### Example for Local Deployment
+
+To use a local Ollama server with the `llama3.2:1b` model, you would configure your `.env` file like this:
+
+```
+LLM_PROVIDER=openai # Using OpenAI compatible API format for local models
+API_KEY=None      # API Key is not needed for local Ollama
+LLM_NAME=llama3.2:1b
+OPENAI_BASE_URL=http://host.docker.internal:11434/v1 # Default Ollama API URL within Docker
+EMBEDDINGS_NAME=huggingface_sentence-transformers/all-mpnet-base-v2 # You can also run embeddings locally if needed
+```
+
+In this case, even though you are using Ollama locally, `LLM_PROVIDER` is set to `openai` because Ollama (and many other local inference engines) are designed to be API-compatible with OpenAI. `OPENAI_BASE_URL` points DocsGPT to the local Ollama server.
+
+## Adding Custom Models (`MODELS_CONFIG_DIR`)
+
+DocsGPT ships with a built-in catalog of models for the providers it
+supports out of the box (OpenAI, Anthropic, Google, Groq, OpenRouter,
+Novita, Hugging Face, DocsGPT). To add **your own
+models** without forking the repo — for example, a Mistral or Together
+account, a self-hosted vLLM endpoint, or any other OpenAI-compatible
+API — point `MODELS_CONFIG_DIR` at a directory of YAML files.
+
+```
+MODELS_CONFIG_DIR=/etc/docsgpt/models
+MISTRAL_API_KEY=sk-...
+```
+
+A minimal YAML for one provider:
+
+```yaml
+# /etc/docsgpt/models/mistral.yaml
+provider: openai_compatible
+display_provider: mistral
+api_key_env: MISTRAL_API_KEY
+base_url: https://api.mistral.ai/v1
+defaults:
+  supports_tools: true
+  context_window: 128000
+models:
+  - id: mistral-large-latest
+    display_name: Mistral Large
+  - id: mistral-small-latest
+    display_name: Mistral Small
+```
+
+After restart, those models appear in `/api/models` and are selectable
+in the UI. A working template lives at
+`application/core/models/examples/mistral.yaml.example`.
+
+**What you can do:**
+
+- Add new `openai_compatible` providers (Mistral, Together, Fireworks,
+  Ollama, vLLM, ...) — one YAML per provider, each with its own
+  `api_key_env` and `base_url`.
+- Extend an existing provider's catalog by dropping a YAML with the
+  same `provider:` value as the built-in (e.g. `provider: anthropic`
+  with extra models).
+- Override a built-in model's capabilities by re-declaring the same
+  `id` — later wins, override is logged at `WARNING`.
+
+**What you cannot do via `MODELS_CONFIG_DIR`:** add a brand-new
+non-OpenAI provider. That requires a Python plugin under
+`application/llm/providers/`. See
+`application/core/models/README.md` for the full schema reference.
+
+### Docker
+
+Mount the directory and set the env var:
+
+```yaml
+# docker-compose.yml
+services:
+  app:
+    image: arc53/docsgpt
+    environment:
+      MODELS_CONFIG_DIR: /etc/docsgpt/models
+      MISTRAL_API_KEY: ${MISTRAL_API_KEY}
+    volumes:
+      - ./my-models:/etc/docsgpt/models:ro
+```
+
+### Misconfiguration
+
+If `MODELS_CONFIG_DIR` is set but the path doesn't exist (or isn't a
+directory), the app logs a `WARNING` at boot and continues with just
+the built-in catalog — it does **not** fail to start. If a YAML
+declares an unknown provider name or has a schema error, the app
+**does** fail to start, with the offending file path in the message.
+
+## Speech-to-Text Settings
+
+DocsGPT can transcribe audio in two places:
+
+- Voice input in the chat.
+- Audio file ingestion. Uploaded `.wav`, `.mp3`, `.m4a`, `.ogg`, and `.webm` files are transcribed first and then passed through the normal parser, chunking, embedding, and indexing pipeline.
+
+The settings below control speech-to-text behaviour for both voice input and audio file ingestion.
+
+| Setting | Purpose | Typical values |
+| --- | --- | --- |
+| `STT_PROVIDER` | Speech-to-text backend provider. | `openai`, `faster_whisper` |
+| `OPENAI_STT_MODEL` | OpenAI transcription model used when `STT_PROVIDER=openai`. | `gpt-4o-mini-transcribe` |
+| `STT_LANGUAGE` | Optional language hint passed to the provider. Leave unset for auto-detection when supported. | `en`, `es`, unset |
+| `STT_MAX_FILE_SIZE_MB` | Maximum file size accepted by the synchronous `/api/stt` endpoint. | `50` |
+| `STT_ENABLE_TIMESTAMPS` | Include timestamp segments in the normalized transcript response and stored parser metadata. | `true`, `false` |
+| `STT_ENABLE_DIARIZATION` | Reserved provider option for speaker diarization. Some providers may ignore it. | `true`, `false` |
+
+### Example: OpenAI Speech-to-Text
+
+```env
+STT_PROVIDER=openai
+OPENAI_API_KEY=YOUR_OPENAI_API_KEY
+OPENAI_STT_MODEL=gpt-4o-mini-transcribe
+STT_LANGUAGE=
+STT_MAX_FILE_SIZE_MB=50
+STT_ENABLE_TIMESTAMPS=false
+STT_ENABLE_DIARIZATION=false
+```
+
+If you already use `API_KEY` for OpenAI, DocsGPT can reuse that key for transcription. Set `OPENAI_API_KEY` only when you want a dedicated key.
+
+### Example: Local `faster_whisper`
+
+```env
+STT_PROVIDER=faster_whisper
+STT_LANGUAGE=en
+STT_ENABLE_TIMESTAMPS=true
+STT_ENABLE_DIARIZATION=false
+```
+
+`faster_whisper` is an optional backend dependency. Install it in the Python environment used by the DocsGPT API and worker before selecting this provider.
+
+## Authentication Settings
+
+DocsGPT includes a JWT (JSON Web Token) based authentication feature for managing sessions or securing local deployments while allowing access.
+
+### `AUTH_TYPE` Overview
+
+The `AUTH_TYPE` setting in your `.env` file or `settings.py` determines the authentication method used by DocsGPT. This allows you to control how users authenticate with your DocsGPT instance.
+
+| Value         | Description                                                                                 |
+| ------------- | ------------------------------------------------------------------------------------------- |
+| `None`        | No authentication is used. Anyone can access the app.                                       |
+| `simple_jwt`  | A single, long-lived JWT token is generated at startup. All requests use this shared token. |
+| `session_jwt` | Unique JWT tokens are generated for each session/user.                                      |
+| `oidc`        | Users sign in through an external OpenID Connect provider (Authentik, Keycloak, Okta, ...). See [SSO with OIDC](/Deploying/OIDC-SSO). |
+
+#### How to Configure
+
+Add the following to your `.env` file (or set in `settings.py`):
+
+```env
+# No authentication (default)
+AUTH_TYPE=None
+
+# OR: Simple JWT (shared token)
+AUTH_TYPE=simple_jwt
+JWT_SECRET_KEY=your_secret_key_here
+
+# OR: Session JWT (per-user/session tokens)
+AUTH_TYPE=session_jwt
+JWT_SECRET_KEY=your_secret_key_here
+
+# OR: SSO via an OpenID Connect provider (Authentik, Keycloak, Okta, ...)
+AUTH_TYPE=oidc
+OIDC_ISSUER=https://auth.example.com/application/o/docsgpt/
+OIDC_CLIENT_ID=your_client_id
+OIDC_FRONTEND_URL=https://docsgpt.example.com
+JWT_SECRET_KEY=your_secret_key_here
+```
+
+- If `AUTH_TYPE` is set to `simple_jwt` or `session_jwt`, a `JWT_SECRET_KEY` is required.
+- If `JWT_SECRET_KEY` is not set, DocsGPT will generate one and store it in `.jwt_secret_key` in the project root.
+
+#### How Each Method Works
+
+- **None**: No authentication. All API and UI access is open.
+- **simple_jwt**:
+  - A single JWT token is generated at startup and printed to the console.
+  - Use this token in the `Authorization` header for all API requests:
+    ```http
+    Authorization: Bearer <SIMPLE_JWT_TOKEN>
+    ```
+  - The frontend will prompt for this token if not already set.
+- **session_jwt**:
+  - Clients can request a new token from `/api/generate_token`.
+  - Use the received token in the `Authorization` header for subsequent requests.
+  - Each user/session gets a unique token.
+- **oidc**:
+  - The frontend redirects users to your identity provider to sign in (OAuth2 Authorization Code + PKCE).
+  - After a successful sign-in, DocsGPT issues its own session JWT; API requests carry it in the `Authorization` header like the other modes.
+  - Stable per-user identities come from the provider — see the full setup guide: [SSO with OIDC](/Deploying/OIDC-SSO).
+  - The same guide covers the optional access controls: group allowlists, silent session renewal, back-channel logout, SCIM provisioning, and login auditing.
+
+#### Security Notes
+
+- Always keep your `JWT_SECRET_KEY` secure and private.
+- If you set it manually, use a strong, random string.
+- If not set, DocsGPT will generate a secure key and persist it in `.jwt_secret_key`.
+
+#### Checking Current Auth Type
+
+- Use the `/api/config` endpoint to check the current `auth_type` and whether authentication is required.
+
+#### Frontend Token Input for `simple_jwt`
+
+If you have configured `AUTH_TYPE=simple_jwt`, the DocsGPT frontend will prompt you to enter the JWT token if it's not already set or is invalid. Paste the `SIMPLE_JWT_TOKEN` (printed to your console when the backend starts) into this field to access the application.
+
+<img
+  src="/jwt-input.png"
+  alt="Frontend prompt for JWT Token"
+  style={{
+    width: "500px",
+    maxWidth: "100%",
+    display: "block",
+    margin: "1em auto",
+  }}
+/>
+
+## S3 Storage Backend
+
+By default DocsGPT stores files locally. Set `STORAGE_TYPE=s3` to use Amazon S3 — or any S3-compatible service (MinIO, Cloudflare R2, Backblaze B2, DigitalOcean Spaces, …) — instead.
+
+| Setting | Description | Default |
+| --- | --- | --- |
+| `STORAGE_TYPE` | `local` or `s3` | `local` |
+| `S3_BUCKET_NAME` | Bucket name | `docsgpt-test-bucket` |
+| `S3_ACCESS_KEY_ID` | Access key ID | — |
+| `S3_SECRET_ACCESS_KEY` | Secret access key | — |
+| `S3_REGION` | Region (use `auto` for Cloudflare R2) | — |
+| `S3_ENDPOINT_URL` | Custom endpoint for S3-compatible services; leave unset for AWS S3 | — |
+| `S3_PATH_STYLE` | Use path-style addressing (required by most non-AWS services) | `false` |
+| `URL_STRATEGY` | `backend` (proxy through API) or `s3` (direct object URLs) | `backend` |
+
+### AWS S3
+
+```env
+STORAGE_TYPE=s3
+S3_BUCKET_NAME=your-bucket-name
+S3_ACCESS_KEY_ID=your-access-key-id
+S3_SECRET_ACCESS_KEY=your-secret-access-key
+S3_REGION=us-east-1
+```
+
+### S3-compatible services (MinIO, Cloudflare R2, …)
+
+Set `S3_ENDPOINT_URL` and usually `S3_PATH_STYLE=true`:
+
+```env
+STORAGE_TYPE=s3
+S3_BUCKET_NAME=your-bucket-name
+S3_ACCESS_KEY_ID=your-access-key-id
+S3_SECRET_ACCESS_KEY=your-secret-access-key
+S3_REGION=auto
+S3_ENDPOINT_URL=https://<account>.r2.cloudflarestorage.com
+S3_PATH_STYLE=true
+```
+
+Your credentials need these permissions on the bucket: `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject`, `s3:ListBucket`, `s3:HeadObject`.
+
+> **Deprecated:** earlier versions reused the `SAGEMAKER_ACCESS_KEY`, `SAGEMAKER_SECRET_KEY`, and `SAGEMAKER_REGION` variables for S3 credentials. These are still honored as a fallback (with a deprecation warning) but you should migrate to the `S3_*` variables above.
+
+## User-Data Storage (Postgres)
+
+DocsGPT stores user data — conversations, agents, prompts, sources, attachments, workflows, logs, and token usage — in **PostgreSQL**. The backend connects via a single setting:
+
+| Setting | Description | Default |
+| --- | --- | --- |
+| `POSTGRES_URI` | SQLAlchemy-compatible Postgres URI. Any standard `postgresql://` form works — DocsGPT normalizes it internally to the `psycopg` v3 dialect. | — |
+| `AUTO_CREATE_DB` | On startup, connect to the server's `postgres` maintenance DB and issue `CREATE DATABASE` if the target is missing. Requires `CREATEDB` or superuser. No-op when the database already exists. Disable in production. | `true` |
+| `AUTO_MIGRATE` | On startup, run `alembic upgrade head` against the target database. Idempotent and serialized across workers via `alembic_version`. Disable in production in favor of an explicit migration step. | `true` |
+
+Example:
+
+```env
+POSTGRES_URI=postgresql://docsgpt:docsgpt@localhost:5432/docsgpt
+# Append ?sslmode=require for managed providers that enforce SSL.
+```
+
+With the defaults, the app applies the schema automatically on first
+boot. To run it explicitly instead (e.g., in CI/CD or a k8s `Job`):
+
+```bash
+python scripts/db/init_postgres.py
+```
+
+The default Docker Compose file bundles a `postgres` service, and the
+app auto-bootstraps the database on boot, so containerized deployments
+need no manual migration step. See
+[PostgreSQL for User Data](/Deploying/Postgres-Migration#production-hardening)
+for the recommended production flow (both flags `false`, migrations
+gated by CI/CD).
+
+<Callout type="info" emoji="ℹ️">
+  `MONGO_URI` is **opt-in**. It is only consulted when you select the
+  MongoDB Atlas vector-store backend (`VECTOR_STORE=mongodb`) or when
+  running the one-shot `scripts/db/backfill.py` migration from a legacy
+  Mongo-based install. Installing the optional Mongo client libraries
+  requires `pip install 'pymongo>=4.6'`. See
+  [PostgreSQL for User Data](/Deploying/Postgres-Migration) for the
+  migration path.
+</Callout>
+
+## Retrieval & RAG Settings
+
+These control how sources are retrieved and whether the advanced RAG features are available. See [Per-Source Configuration](/Sources/Per-source-configuration) and [GraphRAG](/Sources/GraphRAG) for details.
+
+| Setting | Default | Description |
+| --- | --- | --- |
+| `RETRIEVERS_ENABLED` | `["classic", "default"]` | Allow-list of retrievers usable instance-wide. Valid keys: `classic`, `default`, `hybrid`, `graphrag`. A per-source `retriever` must be within this list. |
+| `PER_SOURCE_RETRIEVAL_ENABLED` | `true` | Master switch for per-source retrieval config. When `false`, all sources fall back to the classic retriever regardless of their stored config. |
+| `GRAPHRAG_ENABLED` | `false` | Enable [GraphRAG](/Sources/GraphRAG). Requires `VECTOR_STORE=pgvector`. |
+| `GRAPHRAG_EXTRACTION_MODEL` | unset | Model used for ingest-time graph extraction. Unset reuses the instance default model. |
+| `GRAPHRAG_MAX_CHUNKS_FOR_EXTRACTION` | `2000` | Hard cap on chunks extracted per source (cost control). |
+
+## Embeddings Settings
+
+See [Embeddings](/Models/embeddings) for full guidance.
+
+| Setting | Default | Description |
+| --- | --- | --- |
+| `EMBEDDINGS_NAME` | `huggingface_sentence-transformers/all-mpnet-base-v2` | The embedding model. |
+| `EMBEDDINGS_BASE_URL` | unset | Base URL of a remote OpenAI-compatible embeddings server. Setting it routes all embedding calls there. |
+| `EMBEDDINGS_KEY` | unset | Optional bearer token for the remote embeddings server. |
+| `EMBEDDINGS_MAX_INPUT_TOKENS` | unset | Truncate each remote embedding input to N tokens (guards servers that reject oversized inputs). |
+
+## Tools Settings
+
+| Setting | Default | Description |
+| --- | --- | --- |
+| `DEFAULT_CHAT_TOOLS` | `["memory", "read_webpage", "scheduler"]` | Tools enabled automatically in regular (agentless) chats. See [Tools Basics](/Tools/basics#default-chat-tools). |
+
+## Admin & Access Settings
+
+See [Access Control, Roles & Teams](/Deploying/Access-Control) for the full model.
+
+| Setting | Default | Description |
+| --- | --- | --- |
+| `OIDC_ADMIN_GROUPS` | unset | Comma-separated IdP groups granted the global `admin` role (OIDC only). |
+| `LOCAL_MODE_ADMIN` | `false` | Grants admin in no-auth mode (`AUTH_TYPE=None`) only. **Never enable on a networked deployment.** |
+
+## LLM Provider Settings
+
+| Setting | Default | Description |
+| --- | --- | --- |
+| `OPENAI_RESPONSES_STORE` | `false` | When `true`, allows OpenAI to persist [Responses API](/Models/cloud-providers#openai-responses-api-and-reasoning) state server-side. |
+
+## Realtime Events Settings
+
+The realtime notifications channel has its own settings — see [Realtime Events & Notifications](/Agents/notifications) (`ENABLE_SSE_PUSH`, `EVENTS_STREAM_MAXLEN`, `SSE_MAX_CONCURRENT_PER_USER`, and related).
+
+## Exploring More Settings
+
+These are just the basic settings to get you started. The `settings.py` file contains many more advanced options that you can explore to further customize DocsGPT, such as:
+
+- Vector store configuration (`VECTOR_STORE`, Qdrant, Milvus, LanceDB settings) If you're looking for an easy way to set up a vector store with pgvector, try [Neon](https://get.neon.com/docsgpt).
+- Retriever settings (`RETRIEVERS_ENABLED`)
+- Cache settings (`CACHE_REDIS_URL`)
+- And many more!
+
+For a complete list of available settings and their descriptions, refer to the `settings.py` file in `application/core`. Remember to restart your Docker containers after making changes to your `.env` file or `settings.py` for the changes to take effect.

docs/content/Deploying/Hosting-the-app.mdx

@@ -0,0 +1,33 @@
+import { DeploymentCards } from '../../components/DeploymentCards';
+
+# Deployment Guides
+
+<DeploymentCards
+  items={[
+    {
+      title: 'Amazon Lightsail',
+      link: 'https://docs.docsgpt.cloud/Deploying/Amazon-Lightsail',
+      description: 'Self-hosting DocsGPT on Amazon Lightsail'
+    },
+    {
+      title: 'Railway',
+      link: 'https://docs.docsgpt.cloud/Deploying/Railway',
+      description: 'Hosting DocsGPT on Railway'
+    },
+    {
+      title: 'Civo Compute Cloud',
+      link: 'https://dev.to/rutamhere/deploying-docsgpt-on-civo-compute-c',
+      description: 'Step-by-step guide for Civo deployment'
+    },
+    {
+      title: 'DigitalOcean Droplet', 
+      link: 'https://dev.to/rutamhere/deploying-docsgpt-on-digitalocean-droplet-50ea',
+      description: 'Guide for DigitalOcean deployment'
+    },
+    {
+      title: 'Kamatera Cloud',
+      link: 'https://dev.to/rutamhere/deploying-docsgpt-on-kamatera-performance-cloud-1bj',
+      description: 'Kamatera deployment tutorial'
+    }
+  ]}
+/>

docs/content/Deploying/Kubernetes-Deploying.mdx

@@ -0,0 +1,109 @@
+---
+title: Deploying DocsGPT on Kubernetes
+description: Learn how to self-host DocsGPT on a Kubernetes cluster for scalable and robust deployments.
+---
+
+# Self-hosting DocsGPT
+ on Kubernetes
+
+This guide will walk you through deploying DocsGPT on Kubernetes.
+
+## Prerequisites
+
+Ensure you have the following installed before proceeding:
+
+- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
+- Access to a Kubernetes cluster.
+- [Neon](https://get.neon.com/docsgpt) (optional) for a quick and easy vector store setup with pgvector.
+
+## Folder Structure
+
+The `deployment/k8s` folder contains the necessary deployment and service configuration files:
+
+- `deployments/`
+- `services/`
+- `docsgpt-secrets.yaml`
+
+## Deployment Instructions
+
+1. **Clone the Repository**
+
+   ```sh
+   git clone https://github.com/arc53/DocsGPT.git
+   cd docsgpt/deployment/k8s
+   ```
+
+2. **Configure Secrets (optional)**
+
+   Ensure that you have all the necessary secrets in `docsgpt-secrets.yaml`. Update it with your secrets before applying if you want. By default we will use qdrant as a vectorstore and public docsgpt llm as llm for inference.
+   
+   Alternatively, you can use [Neon](https://get.neon.com/docsgpt) as an easy way to set up your vector store with pgvector, which is highly recommended for quick deployments.
+
+3. **Apply Kubernetes Deployments**
+
+   Deploy your DocsGPT resources using the following commands:
+
+   ```sh
+   kubectl apply -f deployments/
+   ```
+
+4. **Apply Kubernetes Services**
+
+   Set up your services using the following commands:
+
+   ```sh
+   kubectl apply -f services/
+   ```
+
+5. **Apply Secrets**
+
+   Apply the secret configurations:
+
+   ```sh
+   kubectl apply -f docsgpt-secrets.yaml
+   ```
+
+6. **Substitute API URL**
+
+   After deploying the services, you need to update the environment variable `VITE_API_HOST` in your deployment file `deployments/docsgpt-deploy.yaml` with the actual endpoint URL created by your `docsgpt-api-service`.
+
+    ```sh
+    kubectl get services/docsgpt-api-service -o jsonpath='{.status.loadBalancer.ingress[0].ip}' | xargs -I {} sed -i "s|<your-api-endpoint>|{}|g" deployments/docsgpt-deploy.yaml
+    ```
+
+7. **Rerun Deployment**
+
+   After making the changes, reapply the deployment configuration to update the environment variables:
+
+   ```sh
+   kubectl apply -f deployments/
+   ```
+
+## Verifying the Deployment
+
+To verify if everything is set up correctly, you can run the following:
+
+```sh
+kubectl get pods
+kubectl get services
+```
+
+Ensure that the pods are running and the services are available.
+
+## Accessing DocsGPT
+
+To access DocsGPT, you need to find the external IP address of the frontend service. You can do this by running:
+
+```sh
+kubectl get services/docsgpt-frontend-service | awk 'NR>1 {print "http://" $4}'
+```
+
+## Troubleshooting
+
+If you encounter any issues, you can check the logs of the pods for more details:
+
+```sh
+kubectl logs <pod-name>
+```
+
+Replace `<pod-name>` with the actual name of your DocsGPT pod.

docs/content/Deploying/OIDC-SSO.mdx

@@ -0,0 +1,230 @@
+---
+title: SSO with OIDC
+description: Sign users into DocsGPT through any OpenID Connect identity provider (Authentik, Keycloak, Okta, ...) — with group allowlists, silent session renewal, back-channel logout, and SCIM provisioning.
+---
+
+# SSO with OIDC
+
+Setting `AUTH_TYPE=oidc` makes DocsGPT delegate sign-in to an external OpenID Connect identity provider (IdP). Any spec-compliant IdP with a discovery document works; this guide uses [Authentik](https://goauthentik.io/) as the reference provider and includes a short note for Keycloak.
+
+Beyond basic sign-in, this page covers the optional access controls: [group allowlists](#restricting-sign-in-by-group), [silent session renewal](#silent-session-renewal), [back-channel logout](#back-channel-logout), [SCIM user provisioning](#scim-user-provisioning), and [login auditing](#login-auditing).
+
+## How the flow works
+
+1. A user opens DocsGPT without a session. The frontend redirects the browser to `GET /api/auth/oidc/login` on the DocsGPT API.
+2. The backend starts an **OAuth2 Authorization Code + PKCE** flow and redirects to your IdP's sign-in page.
+3. After sign-in, the IdP redirects back to `GET /api/auth/oidc/callback`. The backend exchanges the code server-side, validates the ID token (signature via JWKS, issuer, audience, expiry, nonce), and mints a **DocsGPT session JWT** signed with `JWT_SECRET_KEY`.
+4. The browser returns to your frontend with a short-lived single-use code in the URL fragment; the frontend exchanges it for the session JWT and stores it. From here on, requests are authenticated exactly like the other `AUTH_TYPE` modes (`Authorization: Bearer <token>`).
+5. Signing out clears the local session and redirects through the IdP's end-session endpoint.
+
+The user's identity (`sub` claim by default) becomes the DocsGPT `user_id`, so every user gets their own conversations, sources, agents, and settings.
+
+Sessions last `OIDC_SESSION_LIFETIME_SECONDS` (8 hours by default) and renew without interrupting the user — see [Silent session renewal](#silent-session-renewal).
+
+> Redis must be reachable by the API — it stores the short-lived login state, handoff codes, server-side refresh tokens, and the session revocation denylist. Redis is already a required DocsGPT dependency, so no extra infrastructure is needed.
+
+### IdP compatibility notes
+
+- **Token-endpoint authentication** follows the IdP's discovery document (`token_endpoint_auth_methods_supported`): `client_secret_post` when the IdP advertises it, otherwise HTTP Basic (the RFC default). Okta's default web-app configuration works without extra toggles.
+- **Userinfo fallback**: when the ID token lacks the user-id claim (`OIDC_USER_ID_CLAIM`) — or the groups claim while a group allowlist is configured — the backend fetches the IdP's userinfo endpoint and merges the missing claims. ID-token values win on conflict, and the userinfo `sub` must match the ID token's.
+
+## Settings reference
+
+| Setting | Required | Default | Description |
+| --- | --- | --- | --- |
+| `AUTH_TYPE` | yes | — | Set to `oidc`. |
+| `OIDC_ISSUER` | yes | — | Issuer URL of your IdP. Discovery is read from `<issuer>/.well-known/openid-configuration`. |
+| `OIDC_CLIENT_ID` | yes | — | Client ID registered at the IdP. |
+| `OIDC_FRONTEND_URL` | yes | — | Browser-facing URL of the DocsGPT frontend (where users land after login/logout), e.g. `https://docsgpt.example.com`. |
+| `OIDC_CLIENT_SECRET` | no | — | Set when the IdP client is *confidential*. PKCE is always used, so *public* clients work without a secret. |
+| `OIDC_SCOPES` | no | `openid profile email` | Scopes requested at the IdP. Add `offline_access` when your IdP requires it for refresh tokens (Authentik does). |
+| `OIDC_USER_ID_CLAIM` | no | `sub` | ID-token claim used as the DocsGPT user id. Set to `email` or `preferred_username` for human-readable ids; use `email` when provisioning over [SCIM](#scim-user-provisioning). |
+| `OIDC_REDIRECT_URI` | no | derived | Full callback URL registered at the IdP. Defaults to `<request host>/api/auth/oidc/callback`; set it explicitly when the API runs behind a reverse proxy. |
+| `OIDC_SESSION_LIFETIME_SECONDS` | no | `28800` (8h) | Lifetime of the DocsGPT session JWT. Sessions renew before expiry — see [Silent session renewal](#silent-session-renewal). |
+| `OIDC_PROVIDER_NAME` | no | — | Display name on the sign-in button: `Acme SSO` renders "Sign in with Acme SSO". Unset, the button shows a generic "SSO". |
+| `OIDC_ALLOWED_GROUPS` | no | — | Comma-separated group allowlist. Unset, any authenticated IdP user may sign in — see [Restricting sign-in by group](#restricting-sign-in-by-group). |
+| `OIDC_ADMIN_GROUPS` | no | — | Comma-separated groups whose members are granted the global `admin` role. Re-checked at every login and renewal — see [Granting admin via groups](#granting-admin-via-groups). |
+| `OIDC_GROUPS_CLAIM` | no | `groups` | ID-token/userinfo claim carrying the user's group membership. |
+| `JWT_SECRET_KEY` | recommended | auto-generated | Signs DocsGPT session tokens. Set it explicitly in production — required when running multiple API replicas. |
+
+`SCIM_ENABLED` and `SCIM_TOKEN` are listed in the [SCIM section](#scim-user-provisioning).
+
+## Setting up with Authentik
+
+1. **Create a provider.** In the Authentik admin UI go to **Applications → Providers → Create** and pick **OAuth2/OpenID Provider**:
+   - **Authorization flow**: your preferred flow (e.g. *explicit consent*).
+   - **Client type**: `Public` (no secret, PKCE only) or `Confidential` (also set `OIDC_CLIENT_SECRET` in DocsGPT).
+   - **Redirect URIs**: `https://<your-docsgpt-api>/api/auth/oidc/callback`
+   - **Signing key**: select a certificate so ID tokens are RS256-signed.
+2. **Create an application** (Applications → Applications → Create), link it to the provider, and note its **slug**.
+3. **Find the issuer.** With Authentik's default per-provider issuer mode it is:
+   ```
+   https://<your-authentik-host>/application/o/<application-slug>/
+   ```
+   (the trailing slash is part of the issuer — copy it exactly; the discovery document lives at `.../<application-slug>/.well-known/openid-configuration`).
+4. **Configure DocsGPT** in `.env` and restart:
+   ```env
+   AUTH_TYPE=oidc
+   OIDC_ISSUER=https://auth.example.com/application/o/docsgpt/
+   OIDC_CLIENT_ID=<client id from step 1>
+   # OIDC_CLIENT_SECRET=<only for Confidential client type>
+   OIDC_FRONTEND_URL=https://docsgpt.example.com
+   JWT_SECRET_KEY=<long random string>
+   ```
+
+> Planning to use [silent session renewal](#silent-session-renewal)? Authentik only issues refresh tokens when the `offline_access` scope is requested — set `OIDC_SCOPES=openid profile email offline_access`.
+
+### Which claim becomes the user id?
+
+Authentik's provider setting **Subject mode** controls what lands in the `sub` claim (the default is a hashed user ID — stable but opaque). If you'd rather key DocsGPT users on something readable, either change Subject mode (e.g. *based on username*) or leave Authentik alone and set `OIDC_USER_ID_CLAIM=email` in DocsGPT. Pick one strategy before going live: changing it later gives existing users fresh, empty accounts. If you plan to provision users over [SCIM](#scim-user-provisioning), use `OIDC_USER_ID_CLAIM=email` — SCIM matches users by `userName`, which IdPs typically send as the email.
+
+## Keycloak (and other IdPs)
+
+Any OIDC provider with discovery works the same way. For Keycloak:
+
+```env
+OIDC_ISSUER=https://keycloak.example.com/realms/<realm>
+OIDC_CLIENT_ID=<client id>
+```
+
+Create the client with *Standard flow* enabled and PKCE method `S256`; register the same `/api/auth/oidc/callback` redirect URI.
+
+The feature sections below carry their own per-IdP notes — group claims, refresh tokens, back-channel logout, and SCIM each need one IdP-side setting.
+
+## Restricting sign-in by group
+
+By default any user who can authenticate at the IdP may use DocsGPT. To restrict access to specific IdP groups:
+
+```env
+OIDC_ALLOWED_GROUPS=docsgpt-users,platform-admins
+# OIDC_GROUPS_CLAIM=groups   # only if your IdP uses a different claim name
+```
+
+At login the backend reads the `OIDC_GROUPS_CLAIM` claim (default `groups`) from the ID token, falling back to the userinfo endpoint when the claim is absent. A user whose groups share no entry with the allowlist is rejected with a clean "not authorized" screen (`oidc_error=not_authorized`), and the denial lands in the [audit log](#login-auditing).
+
+Group changes take effect at the next sign-in **or** the next [silent renewal](#silent-session-renewal): whenever the IdP returns a fresh ID token during renewal, the allowlist is re-checked — so removing a user from the allowed group cuts off their session at the next renewal instead of whenever they happen to sign in again.
+
+Getting groups into the token:
+
+- **Authentik** includes group names in the `groups` claim through its default `profile` scope — no extra configuration needed.
+- **Keycloak** does not emit groups by default. On the client, open **Client scopes → the client's dedicated scope → Add mapper → By configuration → Group Membership**, set the claim name to `groups`, and turn **Full group path** off so the claim carries plain names (`devs`) rather than paths (`/devs`).
+
+## Granting admin via groups
+
+Separately from *who may sign in*, you can map an IdP group to the global **admin** role with `OIDC_ADMIN_GROUPS`:
+
+```env
+OIDC_ADMIN_GROUPS=platform-admins
+```
+
+Members of the listed groups are granted admin; the mapping is re-evaluated at every login **and** every [silent renewal](#silent-session-renewal), so removing a user from the admin group revokes their admin at the next renewal (exactly like the sign-in allowlist). It is independent of `OIDC_ALLOWED_GROUPS`, and leaving it unset never mass-revokes admin. On a fresh deployment this is the only way to create the first admin. For the full roles, teams, and admin-dashboard model see [Access Control, Roles & Teams](/Deploying/Access-Control).
+
+## Silent session renewal
+
+The DocsGPT session JWT lives for `OIDC_SESSION_LIFETIME_SECONDS` (default 8 hours). Sessions renew without user-visible interruptions, in one of two ways:
+
+- **With a refresh token.** When the IdP issues one, the backend stores it server-side (in Redis — never in the browser) and the frontend calls `POST /api/auth/oidc/refresh` about 15 minutes before the session expires. The backend redeems the refresh token at the IdP, re-validates the fresh ID token (including the [group allowlist](#restricting-sign-in-by-group)), mints a new session JWT, and rotates the stored refresh token. The user notices nothing.
+- **Without a refresh token.** The frontend lets the session run to expiry and then redirects through the IdP again. While the IdP session is still alive, this round-trip is also silent; the user only sees a sign-in page once the IdP session is gone too.
+
+Getting a refresh token:
+
+- **Keycloak** issues refresh tokens for the authorization-code flow by default — nothing to change.
+- **Authentik** only issues refresh tokens when the `offline_access` scope is requested:
+  ```env
+  OIDC_SCOPES=openid profile email offline_access
+  ```
+
+Revoking the user's consent or sessions at the IdP makes the next renewal fail, and the user must sign in again. For revocation that doesn't wait for the next renewal, configure [back-channel logout](#back-channel-logout).
+
+## Back-channel logout
+
+DocsGPT implements [OIDC Back-Channel Logout 1.0](https://openid.net/specs/openid-connect-backchannel-1_0.html). The IdP POSTs a signed `logout_token` to:
+
+```
+POST https://<your-docsgpt-api>/api/auth/oidc/backchannel-logout
+```
+
+DocsGPT validates the token (signature via JWKS, issuer, audience, replay protection) and immediately revokes the user's live sessions through a Redis denylist — revoked requests get `401` with `error: token_revoked`. Signing the user out at the IdP, or an admin revoking their sessions there, takes effect on their next DocsGPT request instead of at session expiry.
+
+The endpoint is called server-to-server, so it must be reachable from the IdP (it is not a browser redirect).
+
+- **Keycloak**: open the client → **Settings** and set **Backchannel logout URL** to `https://<your-docsgpt-api>/api/auth/oidc/backchannel-logout`.
+- **Authentik** (2025.8.0 and later; marked Preview): on the OAuth2/OpenID provider set **Logout Method** to *Back-channel* and **Logout URI** to the same URL — see the [Authentik logout docs](https://docs.goauthentik.io/add-secure-apps/providers/oauth2/frontchannel_and_backchannel_logout/). Authentik sends the logout token when a user logs out, an admin deletes their session, the account is deactivated, or the session is revoked. On older Authentik versions back-channel logout is unavailable — revocation latency then falls back to the session lifetime, or use [SCIM deactivation](#scim-user-provisioning), which also revokes sessions instantly.
+
+## SCIM user provisioning
+
+DocsGPT exposes a [SCIM 2.0](https://datatracker.ietf.org/doc/html/rfc7644) endpoint so your IdP can drive the user lifecycle: create accounts ahead of first login and — more importantly — deactivate them on offboarding. Deactivating a user revokes their live sessions immediately and blocks future sign-ins (they see an "account disabled" screen); reactivating restores access.
+
+| Setting | Required | Default | Description |
+| --- | --- | --- | --- |
+| `SCIM_ENABLED` | yes | `false` | Set to `true` to serve the `/scim/v2` endpoints. |
+| `SCIM_TOKEN` | yes | — | Bearer token the IdP's SCIM client must present. Use a long random string. |
+
+The base URL is `https://<your-docsgpt-api>/scim/v2`; every request must carry `Authorization: Bearer <SCIM_TOKEN>`.
+
+### Match the SCIM userName to the OIDC user id
+
+SCIM identifies users by `userName`, which DocsGPT matches against its user id — the value of `OIDC_USER_ID_CLAIM`. With the default `sub` claim, the `userName` your IdP sends (typically the email) would never line up with the opaque `sub` of the same user signing in, and DocsGPT would treat them as two unrelated accounts. **When using SCIM, set `OIDC_USER_ID_CLAIM=email` and have the IdP send the email as the SCIM `userName`.**
+
+### What the endpoint supports
+
+| Operation | Support |
+| --- | --- |
+| `GET /scim/v2/ServiceProviderConfig`, `/ResourceTypes`, `/Schemas` | Discovery documents. |
+| `GET /scim/v2/Users` | List, with the exact filter `userName eq "..."` and `startIndex`/`count` pagination (1-based, max 200 per page). |
+| `POST /scim/v2/Users` | Create; returns `409` when the `userName` already exists. |
+| `GET /scim/v2/Users/<id>` | Read. |
+| `PUT` / `PATCH /scim/v2/Users/<id>` | Activate/deactivate via the `active` attribute (Okta's string `"true"`/`"false"` values are accepted). `userName` is immutable; other attributes are ignored. |
+| `DELETE /scim/v2/Users/<id>` | Soft delete — deactivates the account instead of removing data. |
+| `/scim/v2/Groups` | Group provisioning is **not** supported: listing returns an empty result so IdP probes don't fail, and mutations return `501`. Use the [group allowlist](#restricting-sign-in-by-group) for group-based access control instead. |
+
+### IdP setup pointers
+
+- **Okta**: add SCIM provisioning to the app integration with **SCIM connector base URL** = `https://<your-docsgpt-api>/scim/v2` and authentication mode **HTTP Header** carrying the bearer token. Enable creating and deactivating users; skip group push.
+- **Authentik**: create a **SCIM provider** with the same base URL and the token, and attach it to the application as a backchannel provider. Sync users only — leave group mappings out, since DocsGPT answers group provisioning with `501`.
+
+## Login auditing
+
+Authentication activity is recorded in `auth_events`, an append-only Postgres table carrying the user id, event name, IP address, user agent, a JSONB `metadata` column, and a timestamp:
+
+| Event | Recorded when |
+| --- | --- |
+| `oidc_login` | A user signs in successfully. |
+| `oidc_login_denied` | A sign-in is rejected — `metadata.reason` is `not_authorized` (group allowlist) or `account_disabled`. |
+| `oidc_refresh` | A session is silently renewed. |
+| `backchannel_logout` | The IdP revokes sessions via back-channel logout. |
+| `scim_created` / `scim_deactivated` / `scim_reactivated` | SCIM lifecycle changes. |
+| `role_granted` / `role_revoked` | The admin role is granted/revoked (`metadata.source` is `manual` or `oidc_group`). |
+| `admin_user_activated` / `admin_user_deactivated` | An admin activates/deactivates a user. |
+| `admin_sessions_revoked` | An admin force-logs-out a user. |
+| `team.*` | Team management — `team.create`, `team.member_add`, `team.member_role`, `team.member_remove`, `team.share`, `team.unshare`, `team.transfer_owner`, `team.delete`. |
+
+The acting admin is recorded in the event metadata. See [Access Control, Roles & Teams](/Deploying/Access-Control) for the admin and team features that emit these events.
+
+There is no UI for these events yet — query the table directly:
+
+```sql
+SELECT created_at, event, user_id, ip, metadata
+FROM auth_events
+ORDER BY created_at DESC
+LIMIT 50;
+```
+
+## Troubleshooting
+
+When sign-in fails, the browser lands back on the frontend with an `#oidc_error=<code>` fragment and the sign-in screen shows a matching message:
+
+| Code | Cause |
+| --- | --- |
+| `invalid_state` | The login attempt expired (the state is held for 10 minutes) or was replayed. Retrying the sign-in usually fixes it. |
+| `auth_failed` | Token exchange or ID-token validation failed — check the API logs. Most common: `OIDC_ISSUER` doesn't match the issuer the discovery document reports (for Authentik this includes the application slug and trailing slash), or clock skew beyond the allowed 60 seconds. |
+| `missing_claim` | Neither the ID token nor userinfo contains `OIDC_USER_ID_CLAIM`. Make sure the matching scope is requested (`OIDC_SCOPES`) and the IdP actually emits the claim, or switch the setting back to `sub`. |
+| `not_authorized` | The user's groups don't intersect `OIDC_ALLOWED_GROUPS` — see [Restricting sign-in by group](#restricting-sign-in-by-group). |
+| `account_disabled` | The account was deactivated via [SCIM](#scim-user-provisioning) or by an operator. Reactivate it over SCIM to restore access. |
+
+Other issues:
+
+- **IdP shows a redirect URI error** — the callback URL registered at the IdP must match exactly. Behind a reverse proxy, set `OIDC_REDIRECT_URI` to the public callback URL instead of relying on the derived default.
+- **Revoked users can still access DocsGPT** — without back-channel logout, sessions outlive IdP-side revocation until the next renewal or expiry. Configure [back-channel logout](#back-channel-logout) for instant revocation, deactivate the user over [SCIM](#scim-user-provisioning), or lower `OIDC_SESSION_LIFETIME_SECONDS`.
+- **SCIM requests fail** — `404`: `SCIM_ENABLED` is not `true`. `503`: SCIM is enabled but `SCIM_TOKEN` is unset. `401`: the presented bearer token doesn't match `SCIM_TOKEN`.
+- **Login endpoints return 503** — Redis is unreachable or the IdP discovery document can't be fetched from the API host.

docs/content/Deploying/Observability.mdx

@@ -0,0 +1,111 @@
+---
+title: Observability
+description: Send traces, metrics, and logs from DocsGPT to any OpenTelemetry-compatible backend (Axiom, Honeycomb, Grafana, Datadog, Jaeger, etc.).
+---
+
+import { Callout } from 'nextra/components'
+
+# Observability
+
+DocsGPT bundles the OpenTelemetry SDK and auto-instrumentation packages
+in `application/requirements.txt` — they install with the rest of the
+backend deps. Telemetry is **off by default**; opt in by prefixing the
+launch command with `opentelemetry-instrument` and setting OTLP env
+vars.
+
+Auto-instrumentation covers Flask, Starlette, Celery, SQLAlchemy,
+psycopg, Redis, requests, and Python logging. LLM/retriever calls are
+not captured at this layer — see *Going further* below.
+
+## Enabling
+
+Set these env vars in your `.env` (or compose `environment:` block):
+
+```bash
+OTEL_SDK_DISABLED=false
+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
+OTEL_EXPORTER_OTLP_ENDPOINT=https://your-collector.example.com
+OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20<token>
+OTEL_TRACES_EXPORTER=otlp
+OTEL_METRICS_EXPORTER=otlp
+OTEL_LOGS_EXPORTER=otlp
+OTEL_PYTHON_LOG_CORRELATION=true
+OTEL_RESOURCE_ATTRIBUTES=service.name=docsgpt-backend,deployment.environment=prod
+```
+
+Then prefix the process command with `opentelemetry-instrument`. The
+simplest way is a compose override (no image rebuild):
+
+```yaml
+# deployment/docker-compose.override.yaml
+services:
+  backend:
+    command: >
+      opentelemetry-instrument gunicorn -w 1 -k uvicorn_worker.UvicornWorker
+      --bind 0.0.0.0:7091 --config application/gunicorn_conf.py
+      application.asgi:asgi_app
+    environment:
+      - OTEL_SERVICE_NAME=docsgpt-backend
+  worker:
+    command: opentelemetry-instrument celery -A application.app.celery worker -l INFO -B
+    environment:
+      - OTEL_SERVICE_NAME=docsgpt-celery-worker
+```
+
+For local dev, prepend `dotenv run --` so the `OTEL_*` vars from `.env`
+reach `opentelemetry-instrument` before it boots the SDK:
+
+```bash
+dotenv run -- opentelemetry-instrument flask --app application/app.py run --port=7091
+dotenv run -- opentelemetry-instrument celery -A application.app.celery worker -l INFO --pool=solo
+```
+
+
+<Callout type="info" emoji="ℹ️">
+  Logs are exported in-process when `OTEL_LOGS_EXPORTER=otlp` is set —
+  `application/core/logging_config.py` detects the flag and preserves
+  the OTEL log handler. Without it, `logging` writes only to stdout.
+</Callout>
+
+## Backend examples
+
+### Axiom
+
+```bash
+OTEL_EXPORTER_OTLP_ENDPOINT=https://api.axiom.co
+OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20xaat-XXXX,X-Axiom-Dataset=docsgpt
+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
+```
+
+`%20` is the URL-encoded space between `Bearer` and the token. Create
+the dataset in the Axiom UI before sending.
+
+### Self-hosted OTLP collector / Jaeger / Tempo
+
+```bash
+OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
+OTEL_EXPORTER_OTLP_PROTOCOL=grpc
+```
+
+### Honeycomb / Grafana Cloud / Datadog
+
+Each vendor publishes a single-line `OTEL_EXPORTER_OTLP_ENDPOINT` plus
+`OTEL_EXPORTER_OTLP_HEADERS` recipe — drop them in alongside the
+service-name override.
+
+## Caveats
+
+- The Dockerfile uses `gunicorn -w 1`. If you raise worker count, move
+  SDK init into a `post_worker_init` hook to avoid one-thread-per-process
+  exporter contention.
+- `asgi.py` wraps Flask in Starlette's `WSGIMiddleware`. Both
+  instrumentors are installed, so each request produces a Starlette
+  span enclosing a Flask span. Drop
+  `opentelemetry-instrumentation-flask` from `requirements.txt` if the
+  duplication is noisy.
+- OTEL packages add ~50 MB to the image. They install on every build —
+  the runtime cost is zero unless you set `opentelemetry-instrument` on
+  the command and set the OTLP env vars.
+- The OTEL exporter ecosystem currently caps `protobuf` at `<7`, so the
+  backend runs on protobuf 6.x. This will catch up in a future OTEL
+  release.

docs/content/Deploying/Postgres-Migration.mdx

@@ -0,0 +1,151 @@
+---
+title: PostgreSQL for User Data
+description: PostgreSQL is the user-data store for DocsGPT. Covers auto-bootstrap, production hardening, and the one-shot migration from legacy MongoDB deployments.
+---
+
+import { Callout } from 'nextra/components'
+
+# PostgreSQL for User Data
+
+DocsGPT stores conversations, agents, prompts, sources, attachments,
+workflows, logs, and token usage in **PostgreSQL**. MongoDB is no longer
+required.
+
+<Callout type="info" emoji="ℹ️">
+  Vector stores are independent — `VECTOR_STORE` can still be `pgvector`,
+  `faiss`, `qdrant`, `milvus`, `elasticsearch`, or `mongodb`.
+</Callout>
+
+## Quickstart
+
+Three common paths. Each assumes Postgres 13+ and the default env vars
+`AUTO_MIGRATE=true` / `AUTO_CREATE_DB=true` (both ship enabled).
+
+### Docker Compose
+
+The bundled compose file ships a `postgres` service. App boot handles the
+rest — no sidecar, no init job.
+
+```bash
+cd deployment && docker compose up
+```
+
+### Managed Postgres (Neon, RDS, Supabase, Cloud SQL)
+
+Point `POSTGRES_URI` at the provider-given URI. The app applies the
+schema on first boot.
+
+```bash
+export POSTGRES_URI="postgresql://user:pass@host/docsgpt?sslmode=require"
+uvicorn application.asgi:asgi_app --host 0.0.0.0 --port 7091
+```
+
+### Bare-metal Postgres
+
+Run Postgres locally and point `POSTGRES_URI` at the default superuser.
+First boot creates both the database and the schema.
+
+```bash
+export POSTGRES_URI="postgresql://postgres@localhost/docsgpt"
+uvicorn application.asgi:asgi_app --host 0.0.0.0 --port 7091
+```
+
+Prefer a dedicated non-superuser role? Create it once as superuser — the
+app never creates roles.
+
+```sql
+CREATE ROLE docsgpt LOGIN PASSWORD 'docsgpt' CREATEDB;
+-- Then: POSTGRES_URI=postgresql://docsgpt:docsgpt@localhost/docsgpt
+```
+
+## How auto-bootstrap works
+
+Two env vars control startup behavior. Both default to `true` in the
+app and are idempotent.
+
+| Setting | Effect | Requires |
+| --- | --- | --- |
+| `AUTO_CREATE_DB` | If the target database is missing, connects to the server's `postgres` maintenance DB and issues `CREATE DATABASE`. | `CREATEDB` privilege (or superuser) |
+| `AUTO_MIGRATE` | Runs `alembic upgrade head` against the target database. | Table-owner or superuser on the target DB |
+
+Concurrent workers serialize through `alembic_version`, so rolling
+restarts are safe. If the role lacks the required privilege, startup
+fails fast with a clear error rather than silently skipping.
+
+<Callout type="info" emoji="ℹ️">
+  Convenient in dev. In production, disable both and run migrations as
+  an explicit step — see [Production hardening](#production-hardening).
+</Callout>
+
+## Production hardening
+
+Set both flags to `false` in prod and run migrations as a gated,
+auditable step before rolling out the app.
+
+```env
+AUTO_MIGRATE=false
+AUTO_CREATE_DB=false
+```
+
+Run migrations from your CI/CD pipeline, a Kubernetes `Job`, or an
+init-container ahead of the app rollout:
+
+```bash
+python scripts/db/init_postgres.py
+# equivalently:
+alembic -c application/alembic.ini upgrade head
+```
+
+The reasoning: the app's runtime role shouldn't carry DDL privileges,
+migrations should gate each rollout, and an explicit step is
+auditable — implicit first-boot bootstrap is fine for dev but muddies
+prod deploys.
+
+<Callout type="warning" emoji="⚠️">
+  Migrations are not reversible by the app. Always back up production
+  Postgres before running `alembic upgrade head` on a new release.
+</Callout>
+
+## Migrating from MongoDB
+
+One-shot, offline, app stopped. The app itself will create the
+Postgres schema when it boots — you only need to run the data copy.
+
+```bash
+pip install -r application/requirements.txt
+pip install 'pymongo>=4.6'
+
+export POSTGRES_URI="postgresql://docsgpt:docsgpt@localhost:5432/docsgpt"
+export MONGO_URI="mongodb://user:pass@host:27017/docsgpt"
+
+python scripts/db/backfill.py --dry-run    # preview
+python scripts/db/backfill.py              # real run
+# or: python scripts/db/backfill.py --tables users,agents
+```
+
+Then unset `MONGO_URI` and start the backend — nothing consults Mongo
+in the default path anymore. The backfill is idempotent (per-table
+`ON CONFLICT` upserts, event-log tables deduped via `mongo_id`), so
+re-running is safe and re-syncs any drifted rows. Keep Mongo online
+until you've verified Postgres is complete; decommission afterwards
+unless you still use it as a vector store.
+
+<Callout type="warning" emoji="⚠️">
+  No dual-write window and no runtime flag — on the current version,
+  Postgres is the only user-data store the backend reads or writes.
+</Callout>
+
+## Troubleshooting
+
+- **`relation "..." does not exist`** — schema not applied. Either let
+  the app bootstrap it (`AUTO_MIGRATE=true`) or run
+  `python scripts/db/init_postgres.py`.
+- **`permission denied to create database`** — the role lacks
+  `CREATEDB`. As superuser: `ALTER ROLE <name> CREATEDB;`. Or create
+  the database manually and set `AUTO_CREATE_DB=false`.
+- **`role "docsgpt" does not exist`** — roles are never auto-created.
+  As superuser: `CREATE ROLE docsgpt LOGIN PASSWORD '...';`.
+- **SSL errors on a managed provider** — append `?sslmode=require` to
+  `POSTGRES_URI`.
+- **`ModuleNotFoundError: pymongo`** — `pip install 'pymongo>=4.6'`
+  (only needed for the one-shot Mongo backfill).

docs/content/Deploying/Railway.mdx

@@ -0,0 +1,254 @@
+---
+title: Hosting DocsGPT on Railway
+description: Learn how to deploy your own DocsGPT instance on Railway with this step-by-step tutorial
+---
+
+# Self-hosting DocsGPT on Railway
+
+  
+
+Here's a step-by-step guide on how to host DocsGPT on Railway App.
+
+  
+
+At first Clone and set up the project locally to run , test and Modify.
+
+  
+
+### 1. Clone and GitHub SetUp
+
+a. Open Terminal (Windows Shell or Git bash(recommended)).
+
+  
+
+b. Type `git clone https://github.com/arc53/DocsGPT.git`
+
+  
+
+#### Download the package information
+
+  
+
+Once it has finished cloning the repository, it is time to download the package information from all sources. To do so, simply enter the following command:
+
+  
+
+`sudo apt update`
+
+  
+
+#### Install Docker and Docker Compose
+
+  
+
+DocsGPT backend and worker use Python, Frontend is written on React and the whole application is containerized using Docker. To install Docker and Docker Compose, enter the following commands:
+
+  
+
+`sudo apt install docker.io`
+
+  
+
+And now install docker-compose:
+
+  
+
+`sudo apt install docker-compose`
+
+  
+
+#### Access the DocsGPT Folder
+
+  
+
+Enter the following command to access the folder in which the DocsGPT docker-compose file is present.
+
+  
+
+`cd DocsGPT/`
+
+  
+
+#### Prepare the Environment
+
+  
+
+Inside the DocsGPT folder create a `.env` file and copy the contents of `.env_sample` into it.
+
+  
+
+`nano .env`
+
+  
+
+Make sure your `.env` file looks like this:
+
+  
+
+```
+API_KEY=<Your LLM API key>
+LLM_NAME=docsgpt
+VITE_API_STREAMING=true
+```
+
+  
+
+To save the file, press CTRL+X, then Y, and then ENTER.
+
+  
+
+Next, set the correct IP for the Backend by opening the docker-compose.yaml file:
+
+  
+
+`nano deployment/docker-compose.yaml`
+
+  
+
+And Change line 7 to: `VITE_API_HOST=http://localhost:7091`
+
+to this `VITE_API_HOST=http://<your instance public IP>:7091`
+
+  
+
+This will allow the frontend to connect to the backend.
+
+  
+
+#### Running the Application
+
+  
+
+You're almost there! Now that all the necessary bits and pieces have been installed, it is time to run the application. To do so, use the following command:
+
+  
+
+`sudo docker compose -f deployment/docker-compose.yaml up -d`
+
+  
+
+Launching it for the first time will take a few minutes to download all the necessary dependencies and build.
+
+  
+
+Once this is done you can go ahead and close the terminal window.
+
+  
+
+### 2. Pushing it to your own Repository
+
+  
+
+a. Create a Repository on your GitHub.
+
+  
+
+b. Open Terminal in the same directory of the Cloned project.
+
+  
+
+c. Type `git init`
+
+  
+
+d. `git add .`
+
+  
+
+e. `git commit -m "first-commit"`
+
+  
+
+f. `git remote add origin <your  repository  link>`
+
+  
+
+g. `git push git push --set-upstream origin master`
+
+Your local files will now be pushed to your GitHub Account. :)
+  
+
+### 3. Create a Railway Account:
+
+  
+
+If you haven't already, create or log in to your railway account do it by visiting [Railway](https://railway.app/)
+
+  
+
+Signup via **GitHub** [Recommended].
+
+  
+
+### 4. Start New Project:
+
+  
+
+a. Open Railway app and Click on "Start New Project."
+
+  
+
+b. Choose any from the list of options available (Recommended "**Deploy from GitHub Repo**")
+
+  
+
+c. Choose the required Repository from your GitHub.
+
+  
+
+d. Configure and allow access to modify your GitHub content from the pop-up window.
+
+  
+
+e. Agree to all the terms and conditions.
+
+  
+
+PS: It may take a few minutes for the account setup to complete.
+
+  
+
+#### You will get A free trial of $5 (use it for trial and then purchase if satisfied and needed)
+
+  
+
+### 5. Connecting to Your newly Railway app with GitHub
+
+  
+
+a. Choose DocsGPT repo from the list of your GitHub repository that you want to deploy now.
+
+  
+
+b. Click on Deploy now.
+
+  
+
+![Three Tabs will be there](/Railway-selection.png)
+
+  
+
+c. Select Variables Tab.
+
+  
+
+d. Upload the env file here that you used for local setup.
+
+  
+
+e. Go to Settings Tab now.
+
+  
+
+f. Go to "Networking" and click on Generate Domain Name, to get the URL of your hosted project.
+
+  
+
+g. You can update the Root directory, build command, installation command as per need.
+
+*[However recommended not the disturb these options and leave them as default if not that needed.]*
+
+  
+  
+
+Your own DocsGPT is now available at the Generated domain URl. :)

docs/content/Deploying/_meta.js

@@ -0,0 +1,48 @@
+export default {
+  "DocsGPT-Settings": {
+    "title": "⚙️ App Configuration",
+    "href": "/Deploying/DocsGPT-Settings"
+  },
+  "OIDC-SSO": {
+    "title": "🔐 SSO with OIDC",
+    "href": "/Deploying/OIDC-SSO"
+  },
+  "Access-Control": {
+    "title": "👥 Access Control & Teams",
+    "href": "/Deploying/Access-Control"
+  },
+  "Docker-Deploying": {
+    "title": "🛳️ Docker Setup",
+    "href": "/Deploying/Docker-Deploying"
+  },
+  "Development-Environment": {
+    "title": "🛠️Development Environment",
+    "href": "/Deploying/Development-Environment"
+  },
+  "Kubernetes-Deploying": {
+    "title": "☸️ Deploying on Kubernetes",
+    "href": "/Deploying/Kubernetes-Deploying"
+  },
+  "Hosting-the-app": {
+    "title": "☁️ Hosting DocsGPT",
+    "href": "/Deploying/Hosting-the-app"
+  },
+  "Postgres-Migration": {
+    "title": "🐘 PostgreSQL for User Data",
+    "href": "/Deploying/Postgres-Migration"
+  },
+  "Observability": {
+    "title": "🔭 Observability",
+    "href": "/Deploying/Observability"
+  },
+  "Amazon-Lightsail": {
+    "title": "Hosting DocsGPT on Amazon Lightsail",
+    "href": "/Deploying/Amazon-Lightsail",
+    "display": "hidden"
+  },
+  "Railway": {
+    "title": "Hosting DocsGPT on Railway",
+    "href": "/Deploying/Railway",
+    "display": "hidden"
+  }
+}

docs/content/Extensions/Chatwoot-extension.mdx

@@ -0,0 +1,48 @@
+---
+title: Comprehensive Guide to Setting Up the Chatwoot Extension with DocsGPT
+description: This step-by-step guide walks you through the process of setting up the Chatwoot extension with DocsGPT, enabling seamless integration for automated responses and enhanced customer support. Learn how to launch DocsGPT, retrieve your Chatwoot access token, configure the .env file, and start the extension.
+---
+## Chatwoot Extension Setup Guide
+
+### Step 1: Prepare and Start DocsGPT
+
+- **Launch DocsGPT**: Follow the instructions in our [Quickstart](/quickstart) to start DocsGPT. Make sure to load your documentation.
+
+### Step 2: Get Access Token from Chatwoot
+
+- Go to Chatwoot.
+- In your profile settings (located at the bottom left), scroll down and copy the **Access Token**.
+
+### Step 3: Set Up Chatwoot Extension
+
+- Navigate to `/extensions/chatwoot`.
+- Copy the `.env_sample` file and create a new file named `.env`.
+- Fill in the values in the `.env` file as follows:
+
+```env
+docsgpt_url=<Docsgpt_API_URL>
+chatwoot_url=<Chatwoot_URL>
+docsgpt_key=<OpenAI_API_Key or Other_LLM_Key>
+chatwoot_token=<Token from Step 2>
+```
+
+### Step 4: Start the Extension
+
+- Use the command `flask run` to start the extension.
+
+### Step 5: Optional - Extra Validation
+
+- In app.py, uncomment lines 12-13 and 71-75.
+- Add the following lines to your .env file:
+```account_id=(optional) 1
+assignee_id=(optional) 1
+```
+These Chatwoot values help ensure you respond to the correct widget and handle questions assigned to a specific user.
+
+### Stopping Bot Responses for Specific User or Session
+
+- If you want the bot to stop responding to questions for a specific user or session, add a label `human-requested` in your conversation.
+
+### Additional Notes
+
+- For further details on training on other documentation, refer to our [wiki](https://github.com/arc53/DocsGPT/wiki/How-to-train-on-other-documentation).

docs/content/Extensions/_meta.js

@@ -0,0 +1,18 @@
+export default {
+  "api-key-guide": {
+    "title": "🔑 Getting API key",
+    "href": "/Extensions/api-key-guide"
+  },
+  "chat-widget": {
+    "title": "💬️ Chat Widget",
+    "href": "/Extensions/chat-widget"
+  },
+  "search-widget": {
+    "title": "🔎 Search Widget",
+    "href": "/Extensions/search-widget"
+  },
+  "Chatwoot-extension": {
+    "title": "🗣️ Chatwoot Extension",
+    "href": "/Extensions/Chatwoot-extension"
+  }
+}

docs/content/Extensions/api-key-guide.mdx

@@ -0,0 +1,28 @@
+---
+title: API Keys for DocsGPT Integrations
+description: Learn how to obtain, understand, and use DocsGPT API keys to integrate DocsGPT into your external applications and widgets.
+---
+
+# Guide to DocsGPT API Keys
+
+DocsGPT API keys are essential for developers and users who wish to integrate the  DocsGPT models into external applications, such as [our widget](/Extensions/chat-widget). This guide will walk you through the steps of obtaining an API key, starting from uploading your document to understanding the key variables associated with API keys.
+
+## Obtaining Your API Key
+
+After uploading your document, you can obtain an API key either through the graphical user interface or via an API call:
+
+- **Graphical User Interface:** Navigate to the Settings section of the DocsGPT web app, find the Agents option, and press 'Create New' to generate a new agent (which includes an API key).
+- **API Call:** Alternatively, you can use the `/api/create_agent` endpoint to create a new agent. An API key is automatically generated for each agent. For detailed instructions, visit [DocsGPT API Documentation](https://gptcloud.arc53.com/).
+
+## Understanding Key Variables
+
+Upon creating your agent, you will encounter several key variables. Each serves a specific purpose:
+
+- **Name:** Assign a name to your agent for easy identification.
+- **Source:** Indicates the source document(s) linked to your agent, which DocsGPT will use to generate responses.
+- **ID:** A unique identifier for your agent. You can view this by making a call to `/api/get_agents`.
+- **Key:** The API key for the agent, which will be used in your application to authenticate API requests.
+
+With your API key ready, you can now integrate DocsGPT into your application, such as the DocsGPT Widget or any other software, via `/api/answer` or `/stream` endpoints. The source document is preset with the agent, allowing you to bypass fields like `selectDocs` and `active_docs` during implementation.
+
+Congratulations on taking the first step towards enhancing your applications with DocsGPT!

docs/content/Extensions/chat-widget.mdx

@@ -0,0 +1,159 @@
+---
+title: Integrate DocsGPT Chat Widget into Your Web Application
+description: Embed the DocsGPT Widget in your React, HTML, or Nextra projects to provide AI-powered chat functionality to your users.
+---
+import { Tabs } from 'nextra/components'
+
+# Integrating DocsGPT Chat Widget
+
+## Introduction
+
+The DocsGPT Widget is a powerful tool that allows you to integrate AI-driven document assistance directly into your web applications.  This guide will walk you through embedding the DocsGPT Widget into your projects, whether you're using React, plain HTML, or Nextra.  Enhance your user experience by providing seamless access to intelligent document search and chatbot capabilities.
+
+Try out the interactive widget showcase and customize its parameters at the [DocsGPT Widget Demo](https://widget.docsgpt.cloud/).
+
+## Setup
+<Tabs items={['React', 'HTML', 'Nextra']}>
+  <Tabs.Tab>
+
+### Installation
+
+Make sure you have Node.js and npm (or yarn, pnpm) installed in your project.  Navigate to your project directory in the terminal and install the `docsgpt` package:
+
+```bash npm
+npm install docsgpt
+```
+
+### Usage
+
+In your React component file, import the `DocsGPTWidget` component:
+
+```js
+import { DocsGPTWidget } from "docsgpt";
+```
+
+Now, you can embed the widget within your React component's JSX:
+
+```jsx
+<DocsGPTWidget
+  apiHost="https://your-docsgpt-api.com"
+  apiKey=""
+  avatar="https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png"
+  title="Get AI assistance"
+  description="DocsGPT's AI Chatbot is here to help"
+  heroTitle="Welcome to DocsGPT !"
+  heroDescription="This chatbot is built with DocsGPT and utilises GenAI,
+  please review important information using sources."
+  theme="dark"
+  buttonIcon="https://your-icon"
+  buttonBg="#222327"
+/>
+```
+  </Tabs.Tab>
+  <Tabs.Tab>
+
+### Installation
+
+To use the DocsGPT Widget directly in HTML, include the widget script from a CDN in your HTML file:
+
+```html filename="html"
+<script
+  src="https://unpkg.com/docsgpt/dist/legacy/main.js"
+  type="module"
+></script>
+```
+
+### Usage
+
+In your HTML `<body>`, add a `<div>` element where you want to render the widget.  Set an `id` for easy targeting.
+
+```html filename="html"
+<div id="app"></div>
+```
+
+Then, in a `<script type="module">` block, use the `renderDocsGPTWidget` function to initialize the widget, passing the `id` of your `<div>` and a configuration object. To link the widget to your DocsGPT API and specific documents, pass the relevant parameters within the configuration object of `renderDocsGPTWidget`.
+
+```html filename="html"
+<!DOCTYPE html>
+<div id="app"></div>
+<script type="module">
+  window.onload = function() {
+    renderDocsGPTWidget('app', {
+      apiHost: 'http://localhost:7001', // Replace with your API Host
+      apiKey:"",
+      avatar: 'https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png',
+      title: 'Get AI assistance',
+      description: "DocsGPT's AI Chatbot is here to help",
+      heroTitle: 'Welcome to DocsGPT!',
+      heroDescription: 'This chatbot is utilises GenAI, please review important information.',
+      theme:"dark",
+      buttonIcon:"https://your-icon",
+      buttonBg:"#222327"
+    });
+  }
+</script>
+```
+
+  </Tabs.Tab>
+  <Tabs.Tab>
+
+### Installation
+
+Make sure you have Node.js and npm (or yarn, pnpm) installed in your project.  Navigate to your project directory in the terminal and install the `docsgpt` package:
+
+```bash npm
+npm install docsgpt
+```
+
+### Usage with Nextra (Next.js + MDX)
+
+To integrate the DocsGPT Widget into a [Nextra](https://nextra.site/) documentation site (built with Next.js and MDX), create or modify your `pages/_app.js` file as follows:
+
+```js filename="pages/_app.js"
+import { DocsGPTWidget } from "docsgpt";
+
+export default function MyApp({ Component, pageProps }) {
+    return (
+        <>
+            <Component {...pageProps} />
+            <DocsGPTWidget selectDocs="local/docsgpt-sep.zip/"/>
+        </>
+    )
+}
+```
+  </Tabs.Tab>
+</Tabs>
+
+---
+
+## Properties Table
+
+The DocsGPT Widget offers a range of customizable properties that allow you to tailor its appearance and behavior to perfectly match your web application. These parameters can be modified directly when embedding the widget in your React components or HTML code.  Below is a detailed overview of each available prop:
+
+| **Prop**          | **Type**         | **Default Value**                                           | **Description**                                                                                     |
+|--------------------|------------------|-------------------------------------------------------------|-----------------------------------------------------------------------------------------------------|
+| **`apiHost`**      | `string`         | `"https://gptcloud.arc53.com"`                                     | **Required.** The URL of your DocsGPT API backend. This endpoint handles vector search and chatbot queries. |
+| **`apiKey`**       | `string`         | `"your-api-key"`                                            | API key for authentication with your DocsGPT API. Leave empty if no authentication is required.       |
+| **`avatar`**       | `string`         | [`dino-icon-link`](https://d3dg1063dc54p9.cloudfront.net/cute-docsgpt.png)   | URL for the avatar image displayed in the chatbot interface.                                         |
+| **`title`**        | `string`         | `"Get AI assistance"`                                       | Title text shown in the chatbot header.                                                              |
+| **`description`**  | `string`         | `"DocsGPT's AI Chatbot is here to help"`                     | Sub-title or descriptive text displayed below the title in the chatbot header.                       |
+| **`heroTitle`**    | `string`         | `"Welcome to DocsGPT !"`                                    | Welcome message displayed when the chatbot is initially opened.                                     |
+| **`heroDescription`** | `string`     | `"This chatbot is built with DocsGPT and utilises GenAI, please review important information using sources."` | Introductory text providing context or disclaimers about the chatbot.                                 |
+| **`theme`**        | `"dark" \| "light"` | `"dark"`                                                  | Color theme of the widget interface. Options: `"dark"` or `"light"`. Defaults to `"dark"`.         |
+| **`buttonIcon`**   | `string`         | `"https://your-icon"`                                        | URL for the icon image used in the widget's launch button.                                          |
+| **`buttonBg`**     | `string`         | `"#222327"`                                                 | Background color of the widget's launch button.                                                      |
+| **`size`**         | `"small" \| "medium"` | `"medium"`                                              | Size of the widget. Options: `"small"` or `"medium"`. Defaults to `"medium"`.                         |
+| **`showSources`**         | `boolean` | `false`                                              | Enables displaying source URLs for data fetched within the widget. When set to `true`, the widget will show the original sources of the fetched data.                         |
+
+---
+
+## Notes on Widget Properties
+
+*   **Full Customization:**  Every property listed in the table can be customized.  Override the defaults to create a widget that perfectly matches your branding and application context.  From avatars and titles to color schemes, you have fine-grained control over the widget's presentation.
+*   **API Key Handling:** The `apiKey` prop is optional. Only include it if your DocsGPT backend API is configured to require API key authentication. `apiHost` for DocsGPT Cloud is `https://gptcloud.arc53.com/`
+
+## Explore and Customize Further
+
+The DocsGPT Widget is fully open-source, allowing for deep customization and extension beyond the readily available props.
+
+The complete source code for the React-based widget is available in the `extensions/react-widget` directory within the main [DocsGPT GitHub Repository](https://github.com/arc53/DocsGPT).  Feel free to explore the code, fork the repository, and tailor the widget to your exact requirements.

docs/content/Extensions/search-widget.mdx

@@ -0,0 +1,116 @@
+---
+title: Integrate DocsGPT Search Bar into Your Web Application
+description: Embed the DocsGPT Search Bar Widget in your React or HTML projects to provide AI-powered document search functionality to your users.
+---
+import { Tabs } from 'nextra/components'
+
+# Integrating DocsGPT Search Bar Widget
+
+## Introduction
+
+The DocsGPT Search Bar Widget offers a simple yet powerful way to embed AI-powered document search directly into your web applications. This widget allows users to perform searches across your documents or pages, enabling them to quickly find the information they need. This guide will walk you through embedding the Search Bar Widget into your projects, whether you're using React or plain HTML.
+
+Try out the interactive widget showcase and customize its parameters at the [DocsGPT Widget Demo](https://widget.docsgpt.cloud/).
+
+## Setup
+
+<Tabs items={['React', 'HTML']}>
+  <Tabs.Tab>
+## React Setup
+
+### Installation
+
+Make sure you have Node.js and npm (or yarn, pnpm) installed in your project.  Navigate to your project directory in the terminal and install the `docsgpt` package:
+
+```bash npm
+npm install docsgpt
+```
+
+### Usage
+
+In your React component file, import the `SearchBar` component:
+
+```js
+import { SearchBar } from "docsgpt";
+```
+
+Now, you can embed the widget within your React component's JSX:
+
+```jsx
+<SearchBar
+    apiKey="your-api-key"
+    apiHost="https://your-docsgpt-api.com"
+    theme="light"
+    placeholder="Search or Ask AI..."
+    width="300px"
+/>
+```
+  </Tabs.Tab>
+  <Tabs.Tab>
+
+### Installation
+
+To use the DocsGPT Search Bar Widget directly in HTML, include the widget script from a CDN in your HTML file:
+
+```html filename="html"
+<script
+  src="https://unpkg.com/docsgpt/dist/legacy/main.js"
+  type="module"
+></script>
+```
+
+### Usage
+
+In your HTML `<body>`, add a `<div>` element where you want to render the Search Bar Widget. Set an `id` for easy targeting.
+
+```html filename="html"
+<div id="search-bar-container"></div>
+```
+
+Then, in a `<script type="module">` block, use the `renderSearchBar` function to initialize the widget, passing the `id` of your `<div>` and a configuration object. To link the widget to your DocsGPT API and configure its behaviour, pass the relevant parameters within the configuration object of `renderSearchBar`.
+
+```html filename="html"
+<!DOCTYPE html>
+<div id="search-bar-container"></div>
+<script type="module">
+  window.onload = function() {
+    renderSearchBar('search-bar-container', {
+      apiKey: 'your-api-key-here',
+      apiHost: 'https://your-api-host.com',
+      theme: 'light',
+      placeholder: 'Search here...',
+      width: '300px'
+    });
+  }
+</script>
+```
+
+  </Tabs.Tab>
+</Tabs>
+
+---
+
+## Properties Table
+
+The DocsGPT Search Bar Widget offers a range of customizable properties that allow you to tailor its appearance and behavior to perfectly match your web application. These parameters can be modified directly when embedding the widget in your React components or HTML code. Below is a detailed overview of each available prop:
+
+| **Prop**       | **Type**  | **Default Value**                   | **Description**                                                                                  |
+|-----------------|-----------|-------------------------------------|--------------------------------------------------------------------------------------------------|
+| **`apiKey`**    | `string`  | `"your-api-key"`                  | API key for authentication with your DocsGPT API. Leave empty if no authentication is required.                         |
+| **`apiHost`**   | `string`  | `"https://gptcloud.arc53.com"`       | **Required.** The URL of your DocsGPT API backend. This endpoint handles vector similarity search queries.           |
+| **`theme`**     | `"dark" \| "light"` | `"dark"`                            | Color theme of the search bar. Options: `"dark"` or `"light"`. Defaults to `"dark"`.                                     |
+| **`placeholder`** | `string` | `"Search or Ask AI..."`             | Placeholder text displayed in the search input field.                                           |
+| **`width`**     | `string`  | `"256px"`                          | Width of the search bar. Accepts any valid CSS width value (e.g., `"300px"`, `"100%"`, `"20rem"`). |
+
+---
+
+## Notes on Widget Properties
+
+*   **Full Customization:** Every property listed in the table can be customized. Override the defaults to create a Search Bar Widget that perfectly matches your branding and application context.
+*   **API Key Handling:** The `apiKey` prop is optional. Only include it if your DocsGPT backend API is configured to require API key authentication. `apiHost` for DocsGPT Cloud is `https://gptcloud.arc53.com/`
+
+## Explore and Customize Further
+
+The DocsGPT Search Bar Widget is fully open-source, allowing for deep customization and extension beyond the readily available props.
+
+The complete source code for the React-based widget is available in the `extensions/react-widget` directory within the main [DocsGPT GitHub Repository](https://github.com/arc53/DocsGPT). Feel free to explore the code, fork the repository, and tailor the widget to your exact requirements.

docs/content/Guides/Architecture.mdx

@@ -0,0 +1,171 @@
+---
+title: Architecture
+description: High-level architecture of DocsGPT
+---
+
+## Introduction
+
+DocsGPT is designed as a modular and scalable application for knowledge based GenAI system. This document outlines the high-level architecture of DocsGPT, highlighting its key components.
+
+## High-Level Architecture
+
+This diagram provides a bird's-eye view of the DocsGPT architecture, illustrating the main components and their interactions.
+
+```mermaid
+flowchart LR
+    User["User"] --> Frontend["Frontend (React/Vite)"]
+    Frontend --> Backend["Backend API (Flask)"]
+    Backend --> LLM["LLM Integration Layer"] & VectorStore["Vector Stores"] & TaskQueue["Task Queue (Celery)"] & Databases["Databases (Postgres, Redis)"]
+    LLM -- Cloud APIs / Local Engines --> InferenceEngine["Inference Engine"]
+    VectorStore -- Document Embeddings --> Indexes[("Indexes")]
+    TaskQueue -- Asynchronous Tasks --> DocumentIngestion["Document Ingestion"]
+
+    style Frontend fill:#AA00FF,color:#FFFFFF
+    style Backend fill:#AA00FF,color:#FFFFFF
+    style LLM fill:#AA00FF,color:#FFFFFF
+    style TaskQueue fill:#AA00FF,color:#FFFFFF,stroke:#AA00FF
+    style DocumentIngestion fill:#AA00FF,color:#FFFFFF,stroke:none
+```
+
+## Component Descriptions
+
+### 1. Frontend (React/Vite)
+
+*   **Technology:** Built using React and Vite.
+*   **Responsibility:** This is the user interface of DocsGPT, providing users with an UI to ask questions and receive answers, configure prompts, tools and other settings. It handles user input, displays conversation history, shows sources, and manages settings.
+*   **Key Features:**
+    *   Clean and responsive UI.
+    *   Simple static client-side rendering.
+    *   Manages conversation state and settings.
+    *   Communicates with the Backend API for data retrieval and processing.
+
+### 2. Backend API (Flask)
+
+*   **Technology:** Implemented using Flask (Python).
+*   **Responsibility:** The Backend API serves as the core logic and orchestration layer of DocsGPT. It receives requests from the Frontend, Extensions or API clients, processes them, and coordinates interactions between different components.
+*   **Key Features:**
+    *   API endpoints for handling user queries, document uploads, and settings configurations.
+    *   Manages the overall application flow and logic.
+    *   Integrates with the LLM Integration Layer, Vector Stores, Task Queue, Tools, Agents and Databases.
+    *   Provides Swagger documentation for API endpoints.
+
+### 3. LLM Integration Layer (Part of backend)
+
+*   **Technology:** Supports multiple LLM APIs and local engines.
+*   **Responsibility:** This layer provides an abstraction for interacting with Large Language Models (LLMs).
+*   **Key Features:**
+    *   Supports LLMs from OpenAI, Google, Anthropic, Groq, HuggingFace Inference API, Azure OpenAI, also compatible with local models like Ollama, LLaMa.cpp, Text Generation Inference (TGI), SGLang, vLLM, Aphrodite, FriendliAI, and LMDeploy.
+    *   Manages API key handling and request formatting and Tool formatting.
+    *   Offers caching mechanisms to improve response times and reduce API usage.
+    *   Handles streaming responses for a more interactive user experience.
+
+### 4. Vector Stores (Part of backend)
+
+*   **Technology:** Supports multiple vector databases.
+*   **Responsibility:** Vector Stores are used to store and retrieve vector embeddings of document chunks. This enables semantic search and retrieval of relevant document snippets in response to user queries.
+*   **Key Features:**
+    *   Supports vector databases including FAISS, Elasticsearch, Qdrant, Milvus, MongoDB Atlas Vector Search, and pgvector.
+    *   Provides storage and indexing of high-dimensional vector embeddings.
+    *   Enables editing and updating of vector indexes including specific chunks.
+
+### 4a. Retrieval Layer (Part of backend)
+
+*   **Responsibility:** Sits between the API and the vector stores and decides *how* context is fetched for each question. Rather than a single similarity search, a **retriever dispatcher** groups the question's sources by their configured retriever and runs each under a shared token budget, then applies post-retrieval stages before answering.
+*   **Key Features:**
+    *   Multiple retrievers: `classic` (vector similarity), `hybrid` (vector + full-text keyword fusion), and `graphrag` (knowledge-graph / Personalized PageRank).
+    *   Optional query rephrasing before retrieval.
+    *   Optional post-retrieval stages such as LLM pre-screening (map-reduce relevance filtering).
+    *   Per-source behavior via the [per-source configuration](/Sources/Per-source-configuration) contract, so different sources can be chunked and retrieved differently.
+    *   See [GraphRAG](/Sources/GraphRAG) for the knowledge-graph retrieval path.
+
+### 5. Parser Integration Layer (Part of backend)
+
+*   **Technology:** Supports multiple formats for file processing and remote source uploading.
+*   **Responsibility:** Parser Integration Layer handles uploading, parsing, chunking, embedding, and indexing documents.
+*   **Key Features:**
+    *   Supports various document formats (PDF, DOCX, TXT, etc.) and remote sources (web URLs, sitemaps).
+    *   Handles document parsing, text chunking, and embedding generation.
+    *   Utilizes Celery for asynchronous processing, ensuring efficient handling of large documents.
+
+### 6. Task Queue (Celery)
+
+*   **Technology:** Celery with Redis as broker and backend.
+*   **Responsibility:** Celery handles asynchronous task processing, for long-running operations such as document ingestion and indexing. This ensures that the main application remains responsive and efficient.
+*   **Key Features:**
+    *   Manages background tasks for document processing and indexing.
+    *   Improves application responsiveness by offloading heavy tasks.
+    *   Enhances scalability and reliability through distributed task processing.
+
+### 7. Databases (Postgres, Redis)
+
+*   **Technology:** PostgreSQL and Redis.
+*   **Responsibility:** Databases are used for persistent data storage and caching. PostgreSQL stores structured user data such as conversations, agents, prompts, sources, attachments, workflows, logs, token usage, user settings, and API keys. Redis is used as a cache and as the message broker/result backend for Celery.
+*   **Note:** MongoDB is no longer used for user data. It remains an **optional** backend for the vector store (`VECTOR_STORE=mongodb`, i.e. Mongo Atlas Vector Search) and as the source database for the one-shot `scripts/db/backfill.py` migration from legacy installs.
+
+## Request Flow Diagram
+
+This diagram illustrates the sequence of steps involved when a user submits a question to DocsGPT.
+
+```mermaid
+sequenceDiagram
+    participant User
+    participant Frontend
+    participant BackendAPI
+    participant RetrievalLayer
+    participant LLMIntegrationLayer
+    participant VectorStores
+    participant InferenceEngine
+
+    User->>Frontend: User asks a question
+    Frontend->>BackendAPI: API Request (Question)
+    BackendAPI->>RetrievalLayer: Dispatch retrieval (per-source: classic / hybrid / graphrag)
+    RetrievalLayer->>VectorStores: Fetch candidates (vector / keyword / graph)
+    VectorStores-->>RetrievalLayer: Return candidates
+    RetrievalLayer-->>BackendAPI: Ranked, optionally pre-screened chunks
+    BackendAPI->>LLMIntegrationLayer: Send question and document chunks
+    LLMIntegrationLayer->>InferenceEngine: LLM API Request (Prompt + Context)
+    InferenceEngine-->>LLMIntegrationLayer: LLM API Response (Answer)
+    LLMIntegrationLayer-->>BackendAPI: Return Answer
+    BackendAPI->>Frontend: API Response (Answer)
+    Frontend->>User: Display Answer
+
+    Note over Frontend,BackendAPI: Data flow is simplified for clarity
+```
+
+## Deployment Architecture
+
+DocsGPT is designed to be deployed using Docker and Kubernetes, here is a quick overview of a simple k8s deployment.
+
+```mermaid
+graph LR
+    subgraph Kubernetes Cluster
+        subgraph Nodes
+            subgraph Node 1
+                FrontendPod[Frontend Pod]
+                BackendAPIPod[Backend API Pod]
+            end
+            subgraph Node 2
+                CeleryWorkerPod[Celery Worker Pod]
+                RedisPod[Redis Pod]
+            end
+            subgraph Node 3
+                PostgresPod[Postgres Pod]
+                VectorStorePod[Vector Store Pod]
+            end
+        end
+        LoadBalancer[Load Balancer] --> docsgpt-frontend-service[docsgpt-frontend-service]
+        LoadBalancer --> docsgpt-api-service[docsgpt-api-service]
+        docsgpt-frontend-service --> FrontendPod
+        docsgpt-api-service --> BackendAPIPod
+        BackendAPIPod --> CeleryWorkerPod
+        BackendAPIPod --> RedisPod
+        BackendAPIPod --> PostgresPod
+        BackendAPIPod --> VectorStorePod
+        CeleryWorkerPod --> RedisPod
+        BackendAPIPod --> InferenceEngine[(Inference Engine)]
+        VectorStorePod --> Indexes[(Indexes)]
+        PostgresPod --> Data[(Data)]
+        RedisPod --> Cache[(Cache)]
+    end
+    User[User] --> LoadBalancer
+```

docs/content/Guides/Customising-prompts.mdx

@@ -0,0 +1,453 @@
+---
+title: Customizing Prompts
+description: This guide explains how to customize prompts in DocsGPT using the new template-based system with dynamic variable injection.
+---
+
+import Image from 'next/image'
+
+# Customizing Prompts in DocsGPT
+
+Customizing prompts for DocsGPT gives you powerful control over the AI's behavior and responses. With the new template-based system, you can inject dynamic context through organized namespaces, making prompts flexible and maintainable without hardcoding values.
+
+## Quick Start
+
+1. Navigate to `SideBar -> Settings`.
+2. In Settings, select the `Active Prompt` to see various prompt styles.
+3. Click on the `edit icon` on your chosen prompt to customize it.
+
+### Video Demo
+<Image src="/prompts.gif" alt="prompts" width={800} height={500} />
+
+---
+
+## Template-Based Prompt System
+
+DocsGPT now uses **Jinja2 templating** with four organized namespaces for dynamic variable injection:
+
+### Available Namespaces
+
+#### 1. **`system`** - System Metadata
+Access system-level information:
+
+```jinja
+{{ system.date }}         # Current date (YYYY-MM-DD)
+{{ system.time }}         # Current time (HH:MM:SS)
+{{ system.timestamp }}    # ISO 8601 timestamp
+{{ system.request_id }}   # Unique request identifier
+{{ system.user_id }}      # Current user ID
+```
+
+#### 2. **`source`** - Retrieved Documents
+Access RAG (Retrieval-Augmented Generation) document context:
+
+```jinja
+{{ source.content }}      # Concatenated document content
+{{ source.summaries }}    # Alias for content (backward compatible)
+{{ source.documents }}    # List of document objects
+{{ source.count }}        # Number of retrieved documents
+```
+
+#### 3. **`passthrough`** - Request Parameters
+Access custom parameters passed in the API request:
+
+```jinja
+{{ passthrough.company }}     # Custom field from request
+{{ passthrough.user_name }}   # User-provided data
+{{ passthrough.context }}     # Any custom parameter
+```
+
+To use passthrough data, send it in your API request:
+```json
+{
+  "question": "What is the pricing?",
+  "passthrough": {
+    "company": "Acme Corp",
+    "user_name": "Alice",
+    "plan_type": "enterprise"
+  }
+}
+```
+
+#### 4. **`tools`** - Pre-fetched Tool Data
+Access results from tools that run before the agent (like memory tool):
+
+```jinja
+{{ tools.memory.root }}       # Memory tool directory listing
+{{ tools.memory.available }}  # Boolean: is memory available
+```
+
+---
+
+## Example Prompts
+
+### Basic Prompt with Documents
+```jinja
+You are a helpful AI assistant for DocsGPT.
+
+Current date: {{ system.date }}
+
+Use the following documents to answer the question:
+
+{{ source.content }}
+
+Provide accurate, helpful answers with code examples when relevant.
+```
+
+### Advanced Prompt with All Namespaces
+```jinja
+You are an AI assistant for {{ passthrough.company }}.
+
+**System Info:**
+- Date: {{ system.date }}
+- Request ID: {{ system.request_id }}
+
+**User Context:**
+- User: {{ passthrough.user_name }}
+- Role: {{ passthrough.role }}
+
+**Available Documents ({{ source.count }}):**
+{{ source.content }}
+
+**Memory Context:**
+{% if tools.memory.available %}
+{{ tools.memory.root }}
+{% else %}
+No saved context available.
+{% endif %}
+
+Please provide detailed, accurate answers based on the documents above.
+```
+
+### Conditional Logic Example
+```jinja
+You are a DocsGPT assistant.
+
+{% if source.count > 0 %}
+I found {{ source.count }} relevant document(s):
+
+{{ source.content }}
+
+Base your answer on these documents.
+{% else %}
+No documents were found. Please answer based on your general knowledge.
+{% endif %}
+```
+
+---
+
+## Migration Guide
+
+### Legacy Format (Still Supported)
+The old `{summaries}` format continues to work for backward compatibility:
+
+```markdown
+You are a helpful assistant.
+
+Documents:
+{summaries}
+```
+
+This will automatically substitute `{summaries}` with document content.
+
+### New Template Format (Recommended)
+Migrate to the new template syntax for more flexibility:
+
+```jinja
+You are a helpful assistant.
+
+Documents:
+{{ source.content }}
+```
+
+**Migration mapping:**
+- `{summaries}` → `{{ source.content }}` or `{{ source.summaries }}`
+
+---
+
+## Best Practices
+
+### 1. **Use Descriptive Context**
+```jinja
+**Retrieved Documents:**
+{{ source.content }}
+
+**User Query Context:**
+- Company: {{ passthrough.company }}
+- Department: {{ passthrough.department }}
+```
+
+### 2. **Handle Missing Data Gracefully**
+```jinja
+{% if passthrough.user_name %}
+Hello {{ passthrough.user_name }}!
+{% endif %}
+```
+
+### 3. **Leverage Memory for Continuity**
+```jinja
+{% if tools.memory.available %}
+**Previous Context:**
+{{ tools.memory.root }}
+{% endif %}
+
+**Current Question:**
+Please consider the above context when answering.
+```
+
+### 4. **Add Clear Instructions**
+```jinja
+You are a technical support assistant.
+
+**Guidelines:**
+1. Always reference the documents below
+2. Provide step-by-step instructions
+3. Include code examples when relevant
+
+**Reference Documents:**
+{{ source.content }}
+```
+
+---
+
+## Advanced Features
+
+### Looping Over Documents
+```jinja
+{% for doc in source.documents %}
+**Source {{ loop.index }}:** {{ doc.filename }}
+{{ doc.text }}
+
+{% endfor %}
+```
+
+### Date-Based Behavior
+```jinja
+{% if system.date > "2025-01-01" %}
+Note: This is information from 2025 or later.
+{% endif %}
+```
+
+### Custom Formatting
+```jinja
+**Request Information**
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+• Request ID: {{ system.request_id }}
+• User: {{ passthrough.user_name | default("Guest") }}
+• Time: {{ system.time }}
+━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
+```
+
+---
+
+## Tool Pre-Fetching
+
+### Memory Tool Configuration
+Enable memory tool pre-fetching to inject saved context into prompts:
+
+```python
+# In your tool configuration
+{
+  "name": "memory",
+  "config": {
+    "pre_fetch_enabled": true  # Default: true
+  }
+}
+```
+
+Control pre-fetching globally:
+```bash
+# .env file
+ENABLE_TOOL_PREFETCH=true
+```
+
+Or per-request:
+```json
+{
+  "question": "What are the requirements?",
+  "disable_tool_prefetch": false
+}
+```
+
+---
+
+## Debugging Prompts
+
+### View Rendered Prompts in Logs
+Set log level to `INFO` to see the final rendered prompt sent to the LLM:
+
+```bash
+export LOG_LEVEL=INFO
+```
+
+You'll see output like:
+```
+INFO - Rendered system prompt for agent (length: 1234 chars):
+================================================================================
+You are a helpful assistant for Acme Corp.
+
+Current date: 2025-10-30
+Request ID: req_abc123
+
+Documents:
+Technical documentation about...
+================================================================================
+```
+
+### Template Validation
+Test your template syntax before saving:
+```python
+from application.api.answer.services.prompt_renderer import PromptRenderer
+
+renderer = PromptRenderer()
+is_valid = renderer.validate_template("Your prompt with {{ variables }}")
+```
+
+---
+
+## Common Use Cases
+
+### 1. Customer Support Bot
+```jinja
+You are a customer support assistant for {{ passthrough.company }}.
+
+**Customer:** {{ passthrough.customer_name }}
+**Ticket ID:** {{ system.request_id }}
+**Date:** {{ system.date }}
+
+**Knowledge Base:**
+{{ source.content }}
+
+**Previous Interactions:**
+{{ tools.memory.root }}
+
+Please provide helpful, friendly support based on the knowledge base above.
+```
+
+### 2. Technical Documentation Assistant
+```jinja
+You are a technical documentation expert.
+
+**Available Documentation ({{ source.count }} documents):**
+{{ source.content }}
+
+**Requirements:**
+- Provide code examples in {{ passthrough.language }}
+- Focus on {{ passthrough.framework }} best practices
+- Include relevant links when possible
+```
+
+### 3. Internal Knowledge Base
+```jinja
+You are an internal AI assistant for {{ passthrough.department }}.
+
+**Employee:** {{ passthrough.employee_name }}
+**Access Level:** {{ passthrough.access_level }}
+
+**Relevant Documents:**
+{{ source.content }}
+
+Provide detailed answers appropriate for {{ passthrough.access_level }} access level.
+```
+
+---
+
+## Template Syntax Reference
+
+### Variables
+```jinja
+{{ variable_name }}              # Output variable
+{{ namespace.field }}            # Access nested field
+{{ variable | default("N/A") }}  # Default value
+```
+
+### Conditionals
+```jinja
+{% if condition %}
+  Content
+{% elif other_condition %}
+  Other content
+{% else %}
+  Default content
+{% endif %}
+```
+
+### Loops
+```jinja
+{% for item in list %}
+  {{ item.field }}
+{% endfor %}
+```
+
+### Comments
+```jinja
+{# This is a comment and won't appear in output #}
+```
+
+---
+
+## Security Considerations
+
+1. **Input Sanitization**: Passthrough data is automatically sanitized to prevent injection attacks
+2. **Type Filtering**: Only primitive types (string, int, float, bool, None) are allowed in passthrough
+3. **Autoescaping**: Jinja2 autoescaping is enabled by default
+4. **Size Limits**: Consider the token budget when including large documents
+
+---
+
+## Troubleshooting
+
+### Problem: Variables Not Rendering
+**Solution:** Ensure you're using the correct namespace:
+```jinja
+❌ {{ company }}
+✅ {{ passthrough.company }}
+```
+
+### Problem: Empty Output for Tool Data
+**Solution:** Check that tool pre-fetching is enabled and the tool is configured correctly.
+
+### Problem: Syntax Errors
+**Solution:** Validate template syntax. Common issues:
+```jinja
+❌ {{ variable }     # Missing closing brace
+❌ {% if x %         # Missing closing %}
+✅ {{ variable }}
+✅ {% if x %}...{% endif %}
+```
+
+### Problem: Legacy Prompts Not Working
+**Solution:** The system auto-detects template syntax. If your prompt uses `{summaries}`, it will work in legacy mode. To use new features, add `{{ }}` syntax.
+
+---
+
+## API Reference
+
+### Render Prompt via API
+```python
+from application.api.answer.services.prompt_renderer import PromptRenderer
+
+renderer = PromptRenderer()
+rendered = renderer.render_prompt(
+    prompt_content="Your template with {{ passthrough.name }}",
+    user_id="user_123",
+    request_id="req_456",
+    passthrough_data={"name": "Alice"},
+    docs_together="Document content here",
+    tools_data={"memory": {"root": "Files: notes.txt"}}
+)
+```
+
+---
+
+## Conclusion
+
+The new template-based prompt system provides powerful flexibility while maintaining backward compatibility. By leveraging namespaces, you can create dynamic, context-aware prompts that adapt to your specific use case.
+
+**Key Benefits:**
+- ✅ Dynamic variable injection
+- ✅ Organized namespaces
+- ✅ Backward compatible
+- ✅ Security built-in
+- ✅ Easy to debug
+
+Start with simple templates and gradually add complexity as needed. Happy prompting! 🚀

docs/content/Guides/How-to-train-on-other-documentation.mdx

@@ -0,0 +1,48 @@
+---
+title: How to Train on Other Documentation
+description: A step-by-step guide on how to effectively train DocsGPT on additional documentation sources.
+---
+
+import { Callout } from 'nextra/components'
+import Image from 'next/image'
+import { Steps } from 'nextra/components'
+
+## How to train on other documentation
+
+Training on other documentation sources can greatly enhance the versatility and depth of DocsGPT's knowledge. By incorporating diverse materials, you can broaden the AI's understanding and improve its ability to generate insightful responses across a range of topics. Here's a step-by-step guide on how to effectively train DocsGPT on additional documentation sources:
+
+**Get your document ready**:
+
+Make sure you have the document on which you want to train on  ready with you on the device which you are using .You can also use links to the documentation to train on.
+
+<Callout type="warning" emoji="⚠️">
+ Note: Supported file formats include .pdf, .txt, .rst, .docx, .md, .mdx, .csv, .epub, .html, .json, .xlsx, .pptx, .png, .jpg, .jpeg, and audio files (.wav, .mp3, .m4a, .ogg, .webm). You can also train using the link of the documentation.
+
+</Callout>
+
+### Video Demo
+
+<Image src="/docs.gif" alt="prompts" width={800} height={500} />
+
+
+
+<Steps>
+### Step1
+Navigate to the sidebar where you will find `Source Docs` option,here you will find 3 options built in which are default,Web Search and None.
+
+
+### Step 2
+Click on the `Upload icon` just beside the source docs options,now browse and upload the document which you want to train on or select the `remote` option if you have to insert the link of the documentation.
+
+
+### Step 3
+Now you will be able to see the name of the file uploaded under the Uploaded Files ,now click on `Train`,once you click on train it might take some time to train on the document. You will be able to see the `Training progress` and once the training is completed you can click the `finish` button and there you go your document is uploaded.
+
+
+### Step 4
+Go to `New chat`  and from the side bar select the document  you uploaded  under the `Source Docs` and go ahead with your chat, now you can ask questions regarding the document you uploaded and you will get the effective answer based on it.
+
+</Steps>
+
+
+

docs/content/Guides/How-to-use-different-LLM.mdx

@@ -0,0 +1,68 @@
+---
+title:
+description:
+---
+
+import { Callout } from 'nextra/components'
+import Image from 'next/image'
+import { Steps } from 'nextra/components'
+
+# Setting Up Local Language Models for Your App
+
+Setting up local language models for your app can significantly enhance its capabilities, enabling it to understand and generate text in multiple languages without relying on external APIs. By integrating local language models, you can improve privacy, reduce latency, and ensure continuous functionality even in offline environments. Here's a comprehensive guide on how to set up local language models for your application:
+
+## Steps:
+### For cloud version LLM change:
+<Steps >
+### Step 1
+Visit the chat screen and you will be to see the default LLM selected.
+### Step 2
+Click on it and you will get a drop down of various LLM's available to choose.
+### Step 3
+Choose the LLM of your choice.
+
+</Steps>
+
+
+
+
+### Video Demo
+<Image src="/llms.gif" alt="prompts" width={800} height={500} />
+
+### For Open source llm change:
+<Steps>
+### Step 1
+For open source version please edit `LLM_PROVIDER`, `LLM_NAME` and others in the .env file. Refer to [⚙️ App Configuration](/Deploying/DocsGPT-Settings) for more information.
+### Step 2
+Visit [☁️ Cloud Providers](/Models/cloud-providers) for the updated list of online models. Make sure you have the right API_KEY and correct LLM_PROVIDER.
+For self-hosted please visit [🖥️ Local Inference](/Models/local-inference).
+</Steps>
+
+## Fallback LLM
+
+DocsGPT can automatically switch to a fallback LLM when the primary model fails, including mid-stream. This works with both streaming and non-streaming requests.
+
+**Fallback order:**
+1. Per-agent backup models (other models configured on the same agent)
+2. Global fallback (`FALLBACK_LLM_*` env vars below)
+3. Error returned if all fail
+
+| Setting | Description | Default |
+| --- | --- | --- |
+| `FALLBACK_LLM_PROVIDER` | Provider name (e.g., `openai`, `anthropic`, `google`) | — |
+| `FALLBACK_LLM_NAME` | Model name (e.g., `gpt-4o`, `claude-sonnet-4-20250514`) | — |
+| `FALLBACK_LLM_API_KEY` | API key for the fallback provider | Falls back to `API_KEY` |
+
+All three (`FALLBACK_LLM_PROVIDER`, `FALLBACK_LLM_NAME`, and an API key) must resolve for the global fallback to activate.
+
+```env
+FALLBACK_LLM_PROVIDER=anthropic
+FALLBACK_LLM_NAME=claude-sonnet-4-20250514
+FALLBACK_LLM_API_KEY=sk-ant-your-anthropic-key
+```
+
+<Callout type="info">
+For maximum resilience, use a fallback provider from a different cloud than your primary. Each agent can also have multiple models configured — the other models are tried first before the global fallback.
+</Callout>
+
+

docs/content/Guides/Integrations/_meta.js

@@ -0,0 +1,18 @@
+export default {
+  "google-drive-connector": {
+    "title": "🔗 Google Drive",
+    "href": "/Guides/Integrations/google-drive-connector"
+  },
+  "sharepoint-connector": {
+    "title": "🔗 SharePoint / OneDrive",
+    "href": "/Guides/Integrations/sharepoint-connector"
+  },
+  "confluence-connector": {
+    "title": "🔗 Confluence",
+    "href": "/Guides/Integrations/confluence-connector"
+  },
+  "mcp-tool-integration": {
+    "title": "🔗 MCP Tools",
+    "href": "/Guides/Integrations/mcp-tool-integration"
+  }
+}

docs/content/Guides/Integrations/confluence-connector.mdx

@@ -0,0 +1,67 @@
+---
+title: Confluence Connector
+description: Connect your Confluence Cloud workspace as an external knowledge base to upload and process pages directly.
+---
+
+import { Callout } from 'nextra/components'
+import { Steps } from 'nextra/components'
+
+# Confluence Connector
+
+Connect your Confluence Cloud workspace to upload and process pages directly as an external knowledge base. Supports page content and attachments (PDFs, Office files, text files, images, and more). Authentication is handled via Atlassian OAuth 2.0 with automatic token refresh.
+
+## Setup
+
+<Steps>
+
+### Step 1: Create an OAuth 2.0 App in Atlassian
+
+1. Go to [developer.atlassian.com/console/myapps](https://developer.atlassian.com/console/myapps/) and click **Create** > **OAuth 2.0 integration**
+2. Under **Authorization**, add a callback URL:
+   - Local: `http://localhost:7091/api/connectors/callback?provider=confluence`
+   - Production: `https://yourdomain.com/api/connectors/callback?provider=confluence`
+
+### Step 2: Configure Permissions
+
+In your app settings, go to **Permissions** and add the **Confluence API**. Enable these scopes:
+- `read:page:confluence`
+- `read:space:confluence`
+- `read:attachment:confluence`
+
+### Step 3: Get Your Credentials
+
+Go to **Settings** in your app to find the **Client ID** and **Secret**. Copy both.
+
+### Step 4: Configure Environment Variables
+
+Add to your backend `.env` file:
+
+```env
+CONFLUENCE_CLIENT_ID=your-atlassian-client-id
+CONFLUENCE_CLIENT_SECRET=your-atlassian-client-secret
+```
+
+Add to your frontend `.env` file:
+
+```env
+VITE_CONFLUENCE_CLIENT_ID=your-atlassian-client-id
+```
+
+| Variable | Description | Required |
+|----------|-------------|----------|
+| `CONFLUENCE_CLIENT_ID` | Client ID from your Atlassian OAuth app | Yes |
+| `CONFLUENCE_CLIENT_SECRET` | Client secret from your Atlassian OAuth app | Yes |
+| `VITE_CONFLUENCE_CLIENT_ID` | Same Client ID, used by the frontend to show the Confluence option | Yes |
+
+### Step 5: Restart and Use
+
+Restart your application, then go to the upload section in DocsGPT and select **Confluence** as the source. You'll be redirected to Atlassian to sign in, then can browse spaces and select pages to process.
+
+</Steps>
+
+## Troubleshooting
+
+- **Option not appearing** — Verify `VITE_CONFLUENCE_CLIENT_ID` is set in the frontend `.env`, then restart.
+- **Authentication failed** — Check that the callback URL matches exactly, including `?provider=confluence`.
+- **No accessible sites** — Ensure the authenticating user has access to at least one Confluence Cloud site.
+- **Permission denied** — Verify that the Confluence API scopes are enabled in your Atlassian app settings.

docs/content/Guides/Integrations/google-drive-connector.mdx

@@ -0,0 +1,73 @@
+---
+title: Google Drive Connector
+description: Connect your Google Drive as an external knowledge base to upload and process files directly from your Google Drive account.
+---
+
+import { Callout } from 'nextra/components'
+import { Steps } from 'nextra/components'
+
+# Google Drive Connector
+
+Connect your Google Drive account to upload and process files directly as an external knowledge base. Supports Google Workspace files (Docs, Sheets, Slides), Office files, PDFs, text files, CSVs, images, and more. Authentication is handled via Google OAuth 2.0 with automatic token refresh.
+
+## Setup
+
+<Steps>
+
+### Step 1: Create a Google Cloud Project
+
+1. Go to the [Google Cloud Console](https://console.cloud.google.com/) and create a new project (or select an existing one)
+2. Navigate to **APIs & Services** > **Library**, search for "Google Drive API", and click **Enable**
+
+### Step 2: Create OAuth 2.0 Credentials
+
+1. Go to **APIs & Services** > **Credentials** > **Create Credentials** > **OAuth client ID**
+2. If prompted, configure the OAuth consent screen (choose **External**, fill in required fields)
+3. Select **Web application** as the application type
+4. Add your DocsGPT URL to **Authorized JavaScript origins** (e.g. `http://localhost:3000`)
+5. Add your callback URL to **Authorized redirect URIs**:
+   - Local: `http://localhost:7091/api/connectors/callback?provider=google_drive`
+   - Production: `https://yourdomain.com/api/connectors/callback?provider=google_drive`
+6. Click **Create** and copy the **Client ID** and **Client Secret**
+
+### Step 3: Configure Environment Variables
+
+Add to your backend `.env` file:
+
+```env
+GOOGLE_CLIENT_ID=your-google-client-id
+GOOGLE_CLIENT_SECRET=your-google-client-secret
+```
+
+Add to your frontend `.env` file:
+
+```env
+VITE_GOOGLE_CLIENT_ID=your-google-client-id
+```
+
+| Variable | Description | Required |
+|----------|-------------|----------|
+| `GOOGLE_CLIENT_ID` | OAuth Client ID from GCP Credentials | Yes |
+| `GOOGLE_CLIENT_SECRET` | OAuth Client Secret from GCP Credentials | Yes |
+| `VITE_GOOGLE_CLIENT_ID` | Same Client ID, used by the frontend to show the Google Drive option | Yes |
+
+<Callout type="warning" emoji="⚠️">
+Make sure to use the same Google Client ID in both backend and frontend configurations.
+</Callout>
+
+### Step 4: Restart and Use
+
+Restart your application, then go to the upload section in DocsGPT and select **Google Drive** as the source. You'll be redirected to Google to sign in, then can browse and select files to process.
+
+</Steps>
+
+## Troubleshooting
+
+- **Option not appearing** — Verify `VITE_GOOGLE_CLIENT_ID` is set in the frontend `.env`, then restart.
+- **Authentication failed** — Check that the redirect URI matches exactly, including `?provider=google_drive`. Ensure the Google Drive API is enabled.
+- **Permission denied** — Verify the OAuth consent screen is configured and the user has access to the target files.
+- **Files not processing** — Check backend logs and verify that backend environment variables are correctly set.
+
+<Callout type="tip" emoji="💡">
+For production deployments, add your actual domain to the OAuth consent screen and authorized origins/redirect URIs.
+</Callout>

docs/content/Guides/Integrations/mcp-tool-integration.mdx

@@ -0,0 +1,66 @@
+---
+title: MCP Tool Integration
+description: Connect external tools to DocsGPT agents using the Model Context Protocol (MCP) standard.
+---
+
+import { Callout } from 'nextra/components'
+import { Steps } from 'nextra/components'
+
+# MCP Tool Integration
+
+The [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) integration lets you connect external tool servers to DocsGPT. Your agents can then discover and call tools provided by those servers during conversations — for example, querying a CRM, running code, or accessing a database.
+
+## Setup
+
+<Steps>
+
+### Step 1: Configure Environment Variables (Optional)
+
+Only needed if your MCP servers use OAuth authentication:
+
+```env
+MCP_OAUTH_REDIRECT_URI=https://yourdomain.com/api/mcp_server/callback
+```
+
+If not set, falls back to `API_URL/api/mcp_server/callback`.
+
+### Step 2: Add an MCP Server
+
+Go to **Settings** > **Tools** > **Add Tool** > **MCP Server**. Enter the server URL, select an auth type, and click **Test Connection** to verify, then **Save**.
+
+### Step 3: Enable for Your Agent
+
+In your agent configuration, enable the MCP tools you want the agent to use.
+
+</Steps>
+
+## Authentication Types
+
+| Auth Type | Config Fields |
+|-----------|---------------|
+| **None** | — |
+| **Bearer** | `bearer_token` |
+| **API Key** | `api_key`, `api_key_header` (default: `X-API-Key`) |
+| **Basic** | `username`, `password` |
+| **OAuth** | `oauth_scopes` (optional) |
+
+<Callout type="warning">
+For OAuth in production, `MCP_OAUTH_REDIRECT_URI` must be a publicly accessible URL pointing to your DocsGPT backend.
+</Callout>
+
+## API Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/mcp_server/test` | POST | Test a connection without saving |
+| `/api/mcp_server/save` | POST | Save or update a server configuration |
+| `/api/mcp_server/callback` | GET | OAuth callback handler |
+| `/api/mcp_server/oauth_status/<task_id>` | GET | Poll OAuth flow status |
+| `/api/mcp_server/auth_status` | GET | Batch check auth status for all MCP tools |
+
+## Troubleshooting
+
+- **Connection refused** — Verify the URL and that the server is reachable from your backend.
+- **403 Forbidden** — Check credentials and permissions.
+- **Timed out** — Default is 30s; increase timeout in tool config (max 300s).
+- **OAuth "needs_auth" persists** — Verify `MCP_OAUTH_REDIRECT_URI` is correct and Redis is running.

docs/content/Guides/Integrations/sharepoint-connector.mdx

@@ -0,0 +1,63 @@
+---
+title: SharePoint / OneDrive Connector
+description: Connect your Microsoft SharePoint or OneDrive as an external knowledge base to upload and process files directly.
+---
+
+import { Callout } from 'nextra/components'
+import { Steps } from 'nextra/components'
+
+# SharePoint / OneDrive Connector
+
+Connect your SharePoint or OneDrive account to upload and process files directly as an external knowledge base. Supports Office files, PDFs, text files, CSVs, images, and more. Authentication is handled via Microsoft Entra ID (Azure AD) with automatic token refresh.
+
+## Setup
+
+<Steps>
+
+### Step 1: Create an App Registration in Azure
+
+1. Go to the [Azure Portal](https://portal.azure.com/) > **Microsoft Entra ID** > **App registrations** > **New registration**
+2. Set **Redirect URI** (Web) to:
+   - Local: `http://localhost:7091/api/connectors/callback?provider=share_point`
+   - Production: `https://yourdomain.com/api/connectors/callback?provider=share_point`
+
+### Step 2: Configure API Permissions
+
+In your App Registration, go to **API permissions** > **Add a permission** > **Microsoft Graph** > **Delegated permissions** and add: `Files.Read`, `Files.Read.All`, `Sites.Read.All`. Grant admin consent if possible.
+
+### Step 3: Create a Client Secret
+
+Go to **Certificates & secrets** > **New client secret**. Copy the secret value immediately (it won't be shown again).
+
+### Step 4: Configure Environment Variables
+
+Add to your `.env` file:
+
+```env
+MICROSOFT_CLIENT_ID=your-azure-ad-client-id
+MICROSOFT_CLIENT_SECRET=your-azure-ad-client-secret
+MICROSOFT_TENANT_ID=your-azure-ad-tenant-id
+```
+
+| Variable | Description | Required | Default |
+|----------|-------------|----------|---------|
+| `MICROSOFT_CLIENT_ID` | Application (client) ID from App Registration overview | Yes | — |
+| `MICROSOFT_CLIENT_SECRET` | Client secret value | Yes | — |
+| `MICROSOFT_TENANT_ID` | Directory (tenant) ID | No | `common` |
+| `MICROSOFT_AUTHORITY` | Login endpoint override | No | Auto-constructed |
+
+<Callout type="warning">
+`MICROSOFT_TENANT_ID=common` (the default) allows any Microsoft account to authenticate. Set this to your specific tenant ID in production.
+</Callout>
+
+### Step 5: Restart and Use
+
+Restart your application, then go to the upload section in DocsGPT and select **SharePoint / OneDrive** as the source. You'll be redirected to Microsoft to sign in, then can browse and select files to process.
+
+</Steps>
+
+## Troubleshooting
+
+- **Option not appearing** — Verify `MICROSOFT_CLIENT_ID` and `MICROSOFT_CLIENT_SECRET` are set, then restart.
+- **Authentication failed** — Check that the redirect URI matches exactly, including `?provider=share_point`.
+- **Permission denied** — Ensure admin consent is granted and the user has access to the target files.

docs/content/Guides/My-AI-answers-questions-using-external-knowledge.mdx

@@ -0,0 +1,16 @@
+---
+title:
+description:
+---
+
+# Avoiding hallucinations
+
+If your AI uses external knowledge and is not explicit enough, it is ok, because we try to make DocsGPT friendly.
+
+But if you want to adjust it, prompts are now managed through the UI and API using a template-based system. See the [Customising Prompts](/Guides/Customising-prompts) guide for details.
+
+To make the AI stricter about staying on-topic, edit your active prompt template (via **Sidebar → Settings → Active Prompt**) to include instructions like:
+
+```
+If the context provides insufficient information, reply "I cannot answer".
+```

docs/content/Guides/_meta.js

@@ -0,0 +1,35 @@
+export default {
+  "Customising-prompts": {
+    "title": "️💻 Customising Prompts",
+    "href": "/Guides/Customising-prompts"
+  },
+  "How-to-train-on-other-documentation": {
+    "title": "📥 Training on docs",
+    "href": "/Guides/How-to-train-on-other-documentation"
+  },
+  "How-to-use-different-LLM": {
+    "title": "️🤖 How to use different LLM's",
+    "href": "/Guides/How-to-use-different-LLM",
+    "display": "hidden"
+  },
+  "My-AI-answers-questions-using-external-knowledge": {
+    "title": "💭️ Avoiding hallucinations",
+    "href": "/Guides/My-AI-answers-questions-using-external-knowledge",
+    "display": "hidden"
+  },
+  "Architecture": {
+    "title": "🏗️ Architecture",
+    "href": "/Guides/Architecture"
+  },
+  "compression": {
+    "title": "🗜️ Context Compression",
+    "href": "/Guides/compression"
+  },
+  "ocr": {
+    "title": "OCR",
+    "href": "/Guides/ocr"
+  },
+"Integrations": {
+    "title": "🔗 Integrations"
+  }
+}

docs/content/Guides/compression.md

@@ -0,0 +1,37 @@
+# Context Compression
+
+DocsGPT implements a smart context compression system to manage long conversations effectively. This feature prevents conversations from hitting the LLM's context window limit while preserving critical information and continuity.
+
+## How It Works
+
+The compression system operates on a "summarize and truncate" principle:
+
+1.  **Threshold Check**: Before each request, the system calculates the total token count of the conversation history.
+2.  **Trigger**: If the token count exceeds a configured threshold (default: 80% of the model's context limit), compression is triggered.
+3.  **Summarization**: An LLM (potentially a different, cheaper/faster one) processes the older part of the conversation—including previous summaries, user messages, agent responses, and tool outputs.
+4.  **Context Replacement**: The system generates a comprehensive summary of the older history. For subsequent requests, the LLM receives this **Summary + Recent Messages** instead of the full raw history.
+
+### Key Features
+
+*   **Recursive Summarization**: New summaries incorporate previous summaries, ensuring that information from the very beginning of a long chat is not lost.
+*   **Tool Call Support**: The compression logic explicitly handles tool calls and their outputs (e.g., file readings, search results), summarizing their results so the agent retains knowledge of what it has already done.
+*   **"Needle in a Haystack" Preservation**: The prompts are designed to identify and preserve specific, critical details (like passwords, keys, or specific user instructions) even when compressing large amounts of text.
+
+## Configuration
+
+You can configure the compression behavior in your `.env` file or `application/core/settings.py`:
+
+| Setting | Default | Description |
+| :--- | :--- | :--- |
+| `ENABLE_CONVERSATION_COMPRESSION` | `True` | Master switch to enable/disable the feature. |
+| `COMPRESSION_THRESHOLD_PERCENTAGE` | `0.8` | The fraction of the context window (0.0 to 1.0) that triggers compression. |
+| `COMPRESSION_MODEL_OVERRIDE` | `None` | (Optional) Specify a different model ID to use specifically for the summarization task (e.g., using `gpt-3.5-turbo` to compress for `gpt-4`). |
+| `COMPRESSION_MAX_HISTORY_POINTS` | `3` | The number of past compression points to keep in the database (older ones are discarded as they are incorporated into newer summaries). |
+
+## Architecture
+
+The system is modularized into several components:
+
+*   **`CompressionThresholdChecker`**: Calculates token usage and decides when to compress.
+*   **`CompressionService`**: Orchestrates the compression process, manages DB updates, and reconstructs the context (Summary + Recent Messages) for the LLM.
+*   **`CompressionPromptBuilder`**: Constructs the specific prompts used to instruct the LLM to summarize the conversation effectively.

docs/content/Guides/ocr.mdx

@@ -0,0 +1,80 @@
+---
+title: OCR for Sources and Attachments
+description: How OCR works in DocsGPT, how to configure it, and what changes for source ingestion vs chat attachments.
+---
+
+import { Callout } from 'nextra/components'
+
+# Docling OCR for Sources and Attachments
+
+DocsGPT uses Docling as the default parser layer for many document formats. OCR is optional and controlled by two settings:
+
+```env
+DOCLING_OCR_ENABLED=false
+DOCLING_OCR_ATTACHMENTS_ENABLED=false
+```
+
+- `DOCLING_OCR_ENABLED`: OCR behavior for Source Docs ingestion.
+- `DOCLING_OCR_ATTACHMENTS_ENABLED`: OCR behavior for chat attachments uploaded from the message box.
+
+## Processing Flow
+
+### Source Docs flow (Upload and Train)
+
+1. Files are uploaded through `/api/upload`.
+2. Ingestion runs asynchronously in Celery (`ingest_worker`).
+3. `SimpleDirectoryReader` parses files with `get_default_file_extractor`.
+4. For PDFs and image formats, Docling parsers are used. OCR in this path is controlled by `DOCLING_OCR_ENABLED`.
+5. Parsed text is chunked, embedded, and stored in the vector store.
+6. Retrieval during chat uses this indexed text and returns source citations.
+
+### Attachment flow (Chat-only file context)
+
+1. Files are uploaded through `/api/store_attachment`.
+2. Celery task `attachment_worker` parses and stores the attachment in Postgres (`attachments` table).
+3. OCR in this path is controlled by `DOCLING_OCR_ATTACHMENTS_ENABLED`.
+4. Attachments are not vectorized and are not added to the source index.
+5. During answer generation, selected attachment IDs are loaded and passed directly to the LLM pipeline.
+
+## How Docling OCR Works
+
+Docling OCR behavior is different for PDFs vs images:
+
+- PDF parser defaults to hybrid OCR:
+  - text regions: extracted directly
+  - bitmap/image regions: OCR only where needed
+- Image parser defaults to full-page OCR (the whole image is visual content).
+
+By default, Docling parser classes use RapidOCR options (language default: `english`).
+
+<Callout type="info" emoji="ℹ️">
+Parser internals like OCR language and force-full-page OCR are currently set by code defaults, not separate `.env` settings.
+</Callout>
+
+## Attachment Behavior by Model Support
+
+When attachments are used in chat, behavior depends on the selected model/provider:
+
+- If a MIME type is supported, DocsGPT sends files/images through provider-native attachment APIs.
+- If unsupported, DocsGPT falls back to the parsed text content stored for the attachment.
+- For providers that support images but not native PDF attachments, PDF files are converted to images (synthetic PDF support).
+
+This means OCR quality is especially important for text fallback paths and for models without native attachment support.
+
+## Recommended Configuration
+
+For most OCR-enabled use cases, enable both flags:
+
+```env
+DOCLING_OCR_ENABLED=true
+DOCLING_OCR_ATTACHMENTS_ENABLED=true
+```
+
+After changing these settings, restart the API and Celery worker.
+
+## Legacy Fallback Notes
+
+- If Docling is unavailable, DocsGPT falls back to legacy parsers.
+- With OCR disabled, text-based PDFs can still parse, but scanned/image-heavy content may produce little text.
+- For image parsing without Docling OCR, the legacy image parser only extracts text when `PARSE_IMAGE_REMOTE=true`.
+

docs/content/Models/_meta.js

@@ -0,0 +1,14 @@
+export default {
+  "cloud-providers": {
+    "title": "☁️ Cloud Providers",
+    "href": "/Models/cloud-providers"
+  },
+  "local-inference": {
+    "title": "🖥️ Local Inference",
+    "href": "/Models/local-inference"
+  },
+  "embeddings": {
+    "title": "📝 Embeddings",
+    "href": "/Models/embeddings"
+  }
+}

docs/content/Models/cloud-providers.mdx

@@ -0,0 +1,76 @@
+---
+title: Connecting DocsGPT to Cloud LLM Providers
+description: Connect DocsGPT to various Cloud Large Language Model (LLM) providers to power your document Q&A.
+---
+
+# Connecting DocsGPT to Cloud LLM Providers
+
+DocsGPT is designed to seamlessly integrate with a variety of Cloud Large Language Model (LLM) providers, giving you access to state-of-the-art AI models for document question answering.
+
+## Configuration via `.env` file
+
+The primary method for configuring your LLM provider in DocsGPT is through the `.env` file. For a comprehensive understanding of all available settings, please refer to the detailed [DocsGPT Settings Guide](/Deploying/DocsGPT-Settings).
+
+To connect to a cloud LLM provider, you will typically need to configure the following basic settings in your `.env` file:
+
+*   **`LLM_PROVIDER`**:  This setting is essential and identifies the specific cloud provider you wish to use (e.g., `openai`, `google`, `anthropic`).
+*   **`LLM_NAME`**:  Specifies the exact model you want to utilize from your chosen provider (e.g., `gpt-5.1`, `gemini-3.5-flash`, `claude-3-5-sonnet-20241022`).  Refer to your provider's documentation for a list of available models.
+*   **`API_KEY`**:  Almost all cloud LLM providers require an API key for authentication. Obtain your API key from your chosen provider's platform and securely store it in your `.env` file.
+
+## Explicitly Supported Cloud Providers
+
+DocsGPT offers direct, streamlined support for the following cloud LLM providers, making configuration straightforward.  The table below outlines the `LLM_PROVIDER` and example `LLM_NAME` values to use for each provider in your `.env` file.
+
+| Provider                     | `LLM_PROVIDER` | Example `LLM_NAME`          |
+| :--------------------------- | :------------- | :-------------------------- |
+| DocsGPT Public API           | `docsgpt`      | `None`                      |
+| OpenAI                       | `openai`       | `gpt-5.1`                   |
+| OpenAI-compatible (BYOM)     | `openai_compatible` | (any; with per-model `base_url`/`api_key`) |
+| Google (Vertex AI, Gemini)   | `google`       | `gemini-3.5-flash`          |
+| Anthropic (Claude)           | `anthropic`    | `claude-3-5-sonnet-20241022`|
+| Groq                         | `groq`         | `llama-3.3-70b-versatile`   |
+| OpenRouter                   | `openrouter`   | (See OpenRouter docs)       |
+| Novita AI                    | `novita`       | (See Novita docs)           |
+| HuggingFace Inference API    | `huggingface`  | `meta-llama/Llama-3.1-8B-Instruct` |
+| Prem AI                      | `premai`       | (See Prem AI docs)          |
+| AWS SageMaker                | `sagemaker`    | (See SageMaker docs)        |
+
+DocsGPT also ships a **model catalog** (`application/core/models/*.yaml`) that the in-app model picker reads, so common models from these providers — including DeepSeek — appear ready to select once the matching API key is set.
+
+## Connecting to OpenAI-Compatible Cloud APIs
+
+DocsGPT's flexible architecture allows you to connect to any cloud provider that offers an API compatible with the OpenAI API standard. This opens up a vast ecosystem of LLM services.
+
+To connect to an OpenAI-compatible cloud provider, you will still use `LLM_PROVIDER=openai` in your `.env` file.  However, you will also need to specify the API endpoint of your chosen provider using the `OPENAI_BASE_URL` setting.  You will also likely need to provide an `API_KEY` and `LLM_NAME` as required by that provider.
+
+**Example for DeepSeek (OpenAI-Compatible API):**
+
+To connect to DeepSeek, which offers an OpenAI-compatible API, your `.env` file could be configured as follows:
+
+```
+LLM_PROVIDER=openai
+API_KEY=YOUR_API_KEY # Your DeepSeek API key
+LLM_NAME=deepseek-chat # Or your desired DeepSeek model name
+OPENAI_BASE_URL=https://api.deepseek.com/v1 # DeepSeek's OpenAI API URL
+```
+
+Remember to consult the documentation of your chosen OpenAI-compatible cloud provider for their specific API endpoint, required model names, and authentication methods.
+
+### Dedicated `openai_compatible` provider (bring-your-own-model)
+
+Beyond the global `OPENAI_BASE_URL`, DocsGPT has a first-class `openai_compatible` provider. It lets a model carry its **own** `base_url` and `api_key`, which is how per-user "bring your own model" (BYOM) endpoints work — each model can point at a different OpenAI-compatible server without changing instance-wide settings. Outbound requests use an SSRF-pinned HTTP client for safety.
+
+This is the mechanism behind catalog entries like DeepSeek, which declare their own `base_url` and API key environment variable rather than relying on `OPENAI_BASE_URL`.
+
+## OpenAI Responses API and reasoning
+
+For OpenAI models that support it, DocsGPT can call the newer **Responses API** (`/v1/responses`) instead of Chat Completions. This is selected per model in the catalog via an `api_flavor: responses` capability and enables features like server-side reasoning. Related settings:
+
+- `reasoning_effort` — per-model reasoning effort hint (for example `medium`) declared in the model catalog.
+- `OPENAI_RESPONSES_STORE` (default `false`) — when `true`, lets OpenAI persist Responses API state server-side.
+
+See [App Configuration](/Deploying/DocsGPT-Settings) for the full settings reference.
+
+## Adding Support for Other Cloud Providers
+
+If you wish to connect to a cloud provider that is not explicitly listed above or doesn't offer OpenAI API compatibility, you can extend DocsGPT to support it. Within the DocsGPT repository, navigate to the `application/llm` directory. Here, you will find Python files defining the existing LLM integrations.  You can use these files as examples to create a new module for your desired cloud provider.  After creating your new LLM module, you will need to register it within the `llm_creator.py` file. This process involves some coding, but it allows for virtually unlimited extensibility to connect to any cloud-based LLM service with an accessible API.

docs/content/Models/embeddings.md

@@ -0,0 +1,104 @@
+---
+title: Understanding and Configuring Embedding Models in DocsGPT
+description: Learn about embedding models, their importance in DocsGPT, and how to configure them for optimal performance.
+---
+
+# Understanding and Configuring Embedding Models in DocsGPT
+
+Embedding models are a crucial component of DocsGPT, enabling its powerful document understanding and question-answering capabilities. This guide will explain what embedding models are, why they are essential for DocsGPT, and how to configure them.
+
+## What are Embedding Models?
+
+In simple terms, an embedding model is a type of language model that converts text into numerical vectors. These vectors, known as embeddings, capture the semantic meaning of the text.  Think of it as translating words and sentences into a language that computers can understand mathematically, where similar meanings are represented by vectors that are close to each other in vector space.
+
+**Why are embedding models important for DocsGPT?**
+
+DocsGPT uses embedding models for several key tasks:
+
+*   **Semantic Search:** When you upload documents to DocsGPT, the application uses an embedding model to generate embeddings for each document chunk. These embeddings are stored in a vector store. When you ask a question, your query is also converted into an embedding. DocsGPT then performs a semantic search in the vector store, finding document chunks whose embeddings are most similar to your query embedding. This allows DocsGPT to retrieve relevant information based on the *meaning* of your question and documents, not just keyword matching.
+*   **Document Understanding:**  Embeddings help DocsGPT understand the underlying meaning of your documents, enabling it to answer questions accurately and contextually, even if the exact keywords from your question are not present in the retrieved document chunks.
+
+In essence, embedding models are the bridge that allows DocsGPT to understand the nuances of human language and connect your questions to the relevant information within your documents.
+
+## Out-of-the-Box Embedding Model Support in DocsGPT
+
+DocsGPT is designed to be flexible and supports a wide range of embedding models right out of the box:
+
+*   **Sentence Transformers:** DocsGPT supports all models available through the [Sentence Transformers library](https://www.sbert.net/). This library offers a vast selection of pre-trained embedding models, known for their quality and efficiency in various semantic tasks. This is the default (`EMBEDDINGS_NAME=huggingface_sentence-transformers/all-mpnet-base-v2`).
+*   **OpenAI Embeddings:** DocsGPT supports OpenAI embedding models (for example `text-embedding-ada-002`, `text-embedding-3-small`, `text-embedding-3-large`) via the OpenAI API.
+*   **Azure OpenAI Embeddings:** Set `AZURE_EMBEDDINGS_DEPLOYMENT_NAME` alongside your Azure OpenAI configuration.
+*   **Remote OpenAI-compatible Embeddings:** Any server that exposes an OpenAI-compatible `/v1/embeddings` endpoint (for example llama.cpp, vLLM, TEI, or a hosted provider) by setting `EMBEDDINGS_BASE_URL`. See [Remote Embeddings](#remote-openai-compatible-embeddings) below.
+
+## Configuring Sentence Transformer Models
+
+To utilize Sentence Transformer models within DocsGPT, you need to follow these steps:
+
+1.  **Download the Model:** Sentence Transformer models are typically hosted on Hugging Face Model Hub. You need to download your chosen model and place it in the `model/` folder in the root directory of your DocsGPT project.
+
+    For example, to use the `all-mpnet-base-v2` model, you would set `EMBEDDINGS_NAME` as described below, and ensure that the model files are available locally (DocsGPT will attempt to download it if it's not found, but local download is recommended for development and offline use).
+
+2.  **Set `EMBEDDINGS_NAME` in `.env` (or `settings.py`):**  You need to configure the `EMBEDDINGS_NAME` setting in your `.env` file (or `settings.py`) to point to the desired Sentence Transformer model.
+
+    *   **Using a pre-downloaded model from `model/` folder:** You can specify a path to the downloaded model within the `model/` directory. For instance, if you downloaded `all-mpnet-base-v2` and it's in `model/all-mpnet-base-v2`, you could potentially use a relative path like (though direct path to the model name is usually sufficient):
+
+        ```
+        EMBEDDINGS_NAME=huggingface_sentence-transformers/all-mpnet-base-v2
+        ```
+        or simply use the model identifier:
+        ```
+        EMBEDDINGS_NAME=sentence-transformers/all-mpnet-base-v2
+        ```
+
+    *   **Using a model directly from Hugging Face Model Hub:** You can directly specify the model identifier from Hugging Face Model Hub:
+
+        ```
+        EMBEDDINGS_NAME=huggingface_sentence-transformers/all-mpnet-base-v2
+        ```
+
+## Using OpenAI Embeddings
+
+To use OpenAI's `text-embedding-ada-002` embedding model, you need to set `EMBEDDINGS_NAME` to `openai_text-embedding-ada-002` and ensure you have your OpenAI API key configured correctly via `API_KEY` in your `.env` file (if you are not using Azure OpenAI).
+
+**Example `.env` configuration for OpenAI Embeddings:**
+
+```
+LLM_PROVIDER=openai
+API_KEY=YOUR_OPENAI_API_KEY # Your OpenAI API Key
+EMBEDDINGS_NAME=openai_text-embedding-ada-002
+```
+
+## Remote (OpenAI-compatible) Embeddings
+
+If you run your own embedding server, or use a provider that exposes an OpenAI-style embeddings API, point DocsGPT at it with `EMBEDDINGS_BASE_URL`. When this is set, all embedding calls (ingestion and querying) are sent to `{EMBEDDINGS_BASE_URL}/v1/embeddings` in OpenAI format instead of running a local model.
+
+```env
+EMBEDDINGS_BASE_URL=http://localhost:8080   # your OpenAI-compatible embeddings server
+EMBEDDINGS_NAME=your-model-name             # sent as the "model" field in the request
+EMBEDDINGS_KEY=YOUR_API_KEY                 # optional; sent as a Bearer token
+```
+
+- `EMBEDDINGS_BASE_URL` — base URL of the remote server. Setting it switches DocsGPT into remote-embeddings mode.
+- `EMBEDDINGS_NAME` — forwarded as the `model` field in each request.
+- `EMBEDDINGS_KEY` — optional bearer token. If you are using OpenAI directly you can copy `API_KEY` here.
+
+### Guarding against oversized inputs
+
+Some remote servers (notably llama.cpp) reject any single input larger than their physical batch size with a `500` error. Set `EMBEDDINGS_MAX_INPUT_TOKENS` to clip each input to a fixed number of tokens before it is sent:
+
+```env
+EMBEDDINGS_MAX_INPUT_TOKENS=512
+```
+
+When set, each input string is truncated to that many tokens and the overflow is dropped (lossy by design). Token counts use DocsGPT's shared tiktoken encoding, which differs from your server's tokenizer, so choose a limit with some headroom below the server's true limit to absorb tokenizer skew. Leave the setting unset (or `0`) to disable truncation.
+
+## Important: Embedding Dimensions Must Stay Consistent
+
+Each embedding model produces vectors of a fixed dimension, and your vector store is created with that dimension. **Changing `EMBEDDINGS_NAME` to a model with a different dimension is not compatible with an existing index** — FAISS and LanceDB will raise a dimension-mismatch error, and pgvector/Qdrant tables are sized to the original dimension.
+
+If you need to switch embedding models, you must re-ingest your sources so the index is rebuilt with the new dimension. This also applies to the [GraphRAG](/Sources/GraphRAG) graph tables, which are sized to the embedding dimension at creation time.
+
+## Adding Support for Other Embedding Models
+
+If you wish to use an embedding model that is not supported out-of-the-box, a good starting point for adding custom embedding model support is to examine the `base.py` file located in the `application/vectorstore` directory.
+
+Specifically, pay attention to the `EmbeddingsWrapper` and `EmbeddingsSingleton` classes. `EmbeddingsWrapper` provides a way to wrap different embedding model libraries into a consistent interface for DocsGPT. `EmbeddingsSingleton` manages the instantiation and retrieval of embedding model instances. By understanding these classes and the existing embedding model implementations, you can create your own custom integration for virtually any embedding model library you desire.

docs/content/Models/local-inference.mdx

@@ -0,0 +1,89 @@
+---
+title: Connecting DocsGPT to Local Inference Engines
+description: Connect DocsGPT to local inference engines for running LLMs directly on your hardware.
+---
+
+# Connecting DocsGPT to Local Inference Engines
+
+DocsGPT can be configured to leverage local inference engines, allowing you to run Large Language Models directly on your own infrastructure. This approach offers enhanced privacy and control over your LLM processing.
+
+Currently, DocsGPT primarily supports local inference engines that are compatible with the OpenAI API format. This means you can connect DocsGPT to various local LLM servers that mimic the OpenAI API structure.
+
+## Configuration via `.env` file
+
+Setting up a local inference engine with DocsGPT is configured through environment variables in the `.env` file. For a detailed explanation of all settings, please consult the [DocsGPT Settings Guide](/Deploying/DocsGPT-Settings).
+
+To connect to a local inference engine, you will generally need to configure these settings in your `.env` file:
+
+*   **`LLM_PROVIDER`**:  Crucially set this to `openai`. This tells DocsGPT to use the OpenAI-compatible API format for communication, even though the LLM is local.
+*   **`LLM_NAME`**:  Specify the model name as recognized by your local inference engine. This might be a model identifier or left as `None` if the engine doesn't require explicit model naming in the API request.
+*   **`OPENAI_BASE_URL`**:  This is essential. Set this to the base URL of your local inference engine's API endpoint. This tells DocsGPT where to find your local LLM server.
+*   **`API_KEY`**:  Generally, for local inference engines, you can set `API_KEY=None` as authentication is usually not required in local setups.
+
+## Native llama.cpp Support
+
+DocsGPT includes native support for llama.cpp without requiring an OpenAI-compatible server. To use this:
+
+```
+LLM_PROVIDER=llama.cpp
+LLM_NAME=your-model-name
+```
+
+This provider integrates directly with llama.cpp Python bindings.
+
+## Supported Local Inference Engines (OpenAI API Compatible)
+
+DocsGPT is also readily configurable to work with the following local inference engines, all communicating via the OpenAI API format. Here are example `OPENAI_BASE_URL` values for each, based on default setups:
+
+| Inference Engine              | `LLM_PROVIDER` | `OPENAI_BASE_URL`          |
+| :---------------------------- | :------------- | :------------------------- |
+| LLaMa.cpp (server mode)       | `openai`       | `http://localhost:8000/v1`   |
+| Ollama                        | `openai`       | `http://localhost:11434/v1`  |
+| Text Generation Inference (TGI)| `openai`      | `http://localhost:8080/v1`   |
+| SGLang                        | `openai`       | `http://localhost:30000/v1`  |
+| vLLM                          | `openai`       | `http://localhost:8000/v1`   |
+| Aphrodite                     | `openai`       | `http://localhost:2242/v1`   |
+| FriendliAI                    | `openai`       | `http://localhost:8997/v1`   |
+| LMDeploy                      | `openai`       | `http://localhost:23333/v1` |
+
+**Important Note on `localhost` vs `host.docker.internal`:**
+
+The `OPENAI_BASE_URL` examples above use `http://localhost`.  If you are running DocsGPT within Docker and your local inference engine is running on your host machine (outside of Docker), you will likely need to replace `localhost` with `http://host.docker.internal` to ensure Docker can correctly access your host's services. For example, `http://host.docker.internal:11434/v1` for Ollama.
+
+## How the Model Registry Works
+
+DocsGPT uses a **Model Registry** to automatically detect and register available models based on your environment configuration. Understanding this system helps you configure models correctly.
+
+### Automatic Model Detection
+
+When DocsGPT starts, the Model Registry scans your environment variables and automatically registers models from providers that have valid API keys configured:
+
+| Environment Variable   | Provider Models Registered |
+| :--------------------- | :------------------------- |
+| `OPENAI_API_KEY`       | OpenAI models (gpt-5.1, gpt-5-mini, etc.) |
+| `ANTHROPIC_API_KEY`    | Anthropic models (Claude family) |
+| `GOOGLE_API_KEY`       | Google models (Gemini family) |
+| `GROQ_API_KEY`         | Groq models (Llama, Mixtral) |
+| `HUGGINGFACE_API_KEY`  | HuggingFace models |
+
+You can also use the generic `API_KEY` variable with `LLM_PROVIDER` to configure a single provider.
+
+### Custom OpenAI-Compatible Models
+
+When you set `OPENAI_BASE_URL` along with `LLM_PROVIDER=openai` and `LLM_NAME`, the registry automatically creates a custom model entry pointing to your local inference server. This is how local engines like Ollama, vLLM, and others get registered.
+
+### Default Model Selection
+
+The registry determines the default model in this priority order:
+
+1. If `LLM_NAME` is set and matches a registered model, that model becomes the default
+2. Otherwise, the first model from the configured `LLM_PROVIDER` is selected
+3. If neither is set, the first available model in the registry is used
+
+### Multiple Providers
+
+You can configure multiple API keys simultaneously (e.g., both `OPENAI_API_KEY` and `ANTHROPIC_API_KEY`). The registry will load models from all configured providers, giving users the ability to switch between them in the UI.
+
+## Adding Support for Other Local Engines
+
+While DocsGPT currently focuses on OpenAI API compatible local engines, you can extend its capabilities to support other local inference solutions.  To do this, navigate to the `application/llm` directory in the DocsGPT repository.  Examine the existing Python files for examples of LLM integrations. You can create a new module for your desired local engine, and then register it in the `llm_creator.py` file within the same directory. This allows for custom integration with a wide range of local LLM servers beyond those listed above.

docs/content/Sources/GraphRAG.mdx

@@ -0,0 +1,114 @@
+---
+title: GraphRAG — Knowledge-Graph Retrieval
+description: Build a knowledge graph from a source at ingest time and retrieve over it with Personalized PageRank. Covers requirements, enabling, configuration, and the graph view.
+---
+
+import { Callout } from 'nextra/components'
+import Image from 'next/image'
+
+# GraphRAG
+
+GraphRAG augments classic vector retrieval with a **knowledge graph**. During ingestion DocsGPT uses an LLM to extract entities and the relationships between them from a source's chunks, and stores them as a graph alongside the vectors. At query time, a graph retriever uses Personalized PageRank (PPR) to walk that graph from the entities mentioned in your question, surfacing connected context that pure similarity search can miss — useful for multi-hop questions and queries that span related concepts.
+
+<Callout type="warning" emoji="⚠️">
+GraphRAG is **flag-gated** and currently **pgvector-only**. It is available only when both `GRAPHRAG_ENABLED=true` **and** `VECTOR_STORE=pgvector`. On any other vector store the enable action is rejected.
+</Callout>
+
+## Requirements
+
+- A PostgreSQL database with the `pgvector` extension (`VECTOR_STORE=pgvector`). See [PostgreSQL for User Data](/Deploying/Postgres-Migration).
+- `GRAPHRAG_ENABLED=true` in your environment.
+- An LLM configured for extraction (GraphRAG reuses your instance default model unless you override it).
+
+```env
+GRAPHRAG_ENABLED=true
+VECTOR_STORE=pgvector
+```
+
+The graph tables live in the same pgvector database as your embeddings and are sized to the embedding dimension. If you change embedding models you must re-ingest and re-extract (see [Embeddings](/Models/embeddings#important-embedding-dimensions-must-stay-consistent)).
+
+## How it works
+
+1. **Choose GraphRAG** for the source — either at upload time, or by enabling it on an existing source (see below). This sets the source's config to `graphrag` mode.
+2. **Extraction** runs over the source's chunks. For each chunk, the LLM extracts entities and relations, which are written into per-source graph tables. Extraction is durable and resumable via a checkpoint, so it survives restarts and re-runs from scratch each time you re-enable it.
+3. **Query.** Questions against the source are routed to the graph retriever, which runs Personalized PageRank from the query's entities to gather related context.
+
+<Callout type="info" emoji="ℹ️">
+If a source has no graph yet (extraction still running or failed), the graph retriever **falls back to classic vector retrieval** for that source — answers keep working, they just don't use the graph until it is ready.
+</Callout>
+
+## Enabling GraphRAG
+
+### At upload time (recommended)
+
+When you upload a new document, open **Advanced settings** and set **Retriever** to **GraphRAG** (the same dropdown also offers **Hybrid**). The source is created in `graphrag` mode and extraction is enqueued as part of ingestion — no extra step.
+
+<Image
+  src="/graph-rag-settings-before-upload.png"
+  alt="Upload dialog advanced settings showing the Retriever dropdown with Classic, Hybrid, and GraphRAG options"
+  width={661}
+  height={945}
+/>
+
+These are the same [per-source retrieval settings](/Sources/Per-source-configuration) you can change later — choosing the retriever up front just avoids a re-ingest.
+
+### On an existing source
+
+To turn an already-ingested source into a GraphRAG source, use the **Enable GraphRAG** action on the source (it shows a status badge while extraction runs), or call the API:
+
+```bash
+curl -X POST https://your-docsgpt/api/sources/<source_id>/graphrag/enable \
+  -H "Authorization: Bearer <token>"
+```
+
+The response returns a `task_id` for the extraction job:
+
+```json
+{ "success": true, "task_id": "..." }
+```
+
+Notes:
+
+- Requires write access to the source (owner or team `editor`).
+- Returns `400` if GraphRAG isn't available on the workspace (wrong vector store or flag off).
+- Re-running the action rebuilds the graph from scratch rather than no-opping against an existing one.
+- You cannot switch a source to `graphrag` through the [config PATCH endpoint](/Sources/Per-source-configuration#editing-the-config-via-api) — use the upload-time selector or this dedicated endpoint.
+
+## Configuration
+
+Instance-wide settings (see [App Configuration](/Deploying/DocsGPT-Settings)):
+
+| Setting | Default | Description |
+| --- | --- | --- |
+| `GRAPHRAG_ENABLED` | `false` | Master switch for the feature. |
+| `GRAPHRAG_EXTRACTION_MODEL` | `null` | Model used for extraction. `null` reuses the instance default model. |
+| `GRAPHRAG_MAX_CHUNKS_FOR_EXTRACTION` | `2000` | Hard cap on how many chunks are extracted per source (cost control). |
+
+Per-source extraction knobs live under the source config's `graph` object and override the instance defaults:
+
+| Field | Default | Description |
+| --- | --- | --- |
+| `extraction_model` | `null` | Override the extraction model for this source. |
+| `max_chunks` | `null` | Override the chunk cap; `null` falls back to `GRAPHRAG_MAX_CHUNKS_FOR_EXTRACTION`. |
+| `gleanings` | `0` | Extra extraction passes per chunk to catch entities missed on the first pass. Off by default (each pass costs additional LLM calls). |
+
+<Callout type="warning" emoji="⚠️">
+Graph extraction makes an LLM call per chunk (more if `gleanings > 0`), so it has a real token cost. The cost is attributed to token usage under a `graph_extraction` tag, and the `max_chunks` cap bounds it.
+</Callout>
+
+## Visualizing the graph
+
+GraphRAG sources expose a **graph view** in the UI — an interactive network of the extracted entities and relationships. It is backed by two read endpoints:
+
+```text
+GET /api/sources/<source_id>/graph                    # bounded {nodes, edges} overview
+GET /api/sources/<source_id>/graph/node/<node_id>     # one node and its neighbors
+```
+
+The overview is bounded to a default node limit to keep large graphs responsive.
+
+## Related
+
+- [Per-Source Configuration](/Sources/Per-source-configuration) — the config object GraphRAG plugs into.
+- [PostgreSQL for User Data](/Deploying/Postgres-Migration) — required pgvector setup.
+- [Embeddings](/Models/embeddings) — embedding-dimension constraints that also apply to the graph tables.

docs/content/Sources/Per-source-configuration.mdx

@@ -0,0 +1,177 @@
+---
+title: Per-Source Configuration (Chunking & Retrieval)
+description: Tune how each knowledge source is chunked at ingest time and retrieved at query time — chunking strategy, retriever, exposure, top-k, pre-screening and more.
+---
+
+import { Callout } from 'nextra/components'
+import Image from 'next/image'
+
+# Per-Source Configuration
+
+Every source in DocsGPT carries its own **behavior contract** — a small config object that controls how that source is *chunked* when it is ingested and how it is *retrieved* when you ask a question. This lets you tune each source independently: a large reference manual can use a different chunking strategy and retriever than a short FAQ.
+
+You edit this config from a source's settings in the UI (shown below), or through the API. The same options are also available in **Advanced settings** when you first upload a document.
+
+<Image
+  src="/sources-settings-screen.png"
+  alt="Source settings panel showing Retrieval options (retriever, top-k, score threshold, rephrase, exposure, prescreen) and Chunking options (strategy, max/min tokens, duplicate headers)"
+  width={633}
+  height={862}
+/>
+
+<Callout type="info" emoji="ℹ️">
+Per-source retrieval is enabled by default. Operators can turn it off instance-wide with `PER_SOURCE_RETRIEVAL_ENABLED=false`, in which case all sources fall back to the classic retriever regardless of their stored config.
+</Callout>
+
+## Two kinds of settings: live vs. bake-time
+
+The config has two groups of settings that differ in *when* they take effect:
+
+| Group | When it applies | Re-ingest needed? |
+| --- | --- | --- |
+| **Retrieval** (`retrieval.*`) | Query time — applied live on the next question | No |
+| **Chunking** (`chunking.*`) | Ingest time — baked into the stored chunks | **Yes** |
+
+Changing a retrieval setting takes effect immediately. Changing a chunking setting only affects documents ingested *after* the change, so you must re-ingest the source to apply it to existing content. The API response includes a `requires_reingest` flag to make this explicit.
+
+## Chunking configuration
+
+Chunking decides how a document is split into the pieces that get embedded and stored.
+
+```json
+{
+  "chunking": {
+    "strategy": "classic_chunk",
+    "max_tokens": 1250,
+    "min_tokens": 150,
+    "duplicate_headers": false
+  }
+}
+```
+
+| Field | Default | Description |
+| --- | --- | --- |
+| `strategy` | `classic_chunk` | Which chunking algorithm to use (see below). |
+| `max_tokens` | `1250` | Upper bound on chunk size in tokens. |
+| `min_tokens` | `150` | Lower bound; small fragments are merged up to this size. |
+| `duplicate_headers` | `false` | Repeat section headers into each child chunk for context. |
+
+### Available chunking strategies
+
+| Strategy | Behavior |
+| --- | --- |
+| `classic_chunk` | The default token-window splitter. An empty config reproduces DocsGPT's historical chunking byte-for-byte. |
+| `recursive` | Recursive character/token splitter that tries to break on natural boundaries (paragraphs, sentences). |
+| `markdown` | Splits along Markdown structure (headings, sections) — good for docs and wikis. |
+| `parent_child` | Embeds small **child** chunks for precise matching but carries a larger **parent** window in metadata, so the model still sees surrounding context. |
+| `semantic` | Embeds sentences and splits where meaning shifts (at the 95th-percentile cosine-distance gap between adjacent sentences), falling back to `recursive` on failure. Produces topically coherent chunks at the cost of extra embedding calls during ingest. |
+
+<Callout type="warning" emoji="⚠️">
+Chunking is bake-time. After changing `strategy`, `max_tokens`, `min_tokens`, or `duplicate_headers`, re-ingest the source so existing chunks are rebuilt.
+</Callout>
+
+## Retrieval configuration
+
+Retrieval decides which chunks are pulled in to answer a question. These settings apply live.
+
+```json
+{
+  "retrieval": {
+    "retriever": "classic",
+    "exposure": "prefetch",
+    "chunks": 2,
+    "score_threshold": null,
+    "rephrase_query": true,
+    "prescreen": null
+  }
+}
+```
+
+| Field | Default | Description |
+| --- | --- | --- |
+| `retriever` | `classic` | Retrieval strategy: `classic`, `hybrid`, or `graphrag`. |
+| `exposure` | `prefetch` | How retrieved context reaches the model: `prefetch` or `agentic_tool` (see below). |
+| `chunks` | `2` | Final number of chunks (top-k) returned to the answer. Range 1–500. |
+| `score_threshold` | `null` | Minimum similarity score. Honored by pgvector and MongoDB Atlas; other stores ignore it. |
+| `rephrase_query` | `true` | Whether to run a query-rephrasing side-call before retrieval. |
+| `prescreen` | `null` | Optional LLM relevance filter (see below). `null` = off. |
+
+### Retrievers
+
+- **`classic`** — Vector similarity search. The default and a safe choice for any vector store.
+- **`hybrid`** — Fuses vector search with full-text keyword search using Reciprocal Rank Fusion, which improves recall for exact terms, codes, and names that pure vector search can miss.
+- **`graphrag`** — Knowledge-graph retrieval. Set indirectly when you enable GraphRAG on a source. See [GraphRAG](/Sources/GraphRAG).
+
+<Callout type="warning" emoji="⚠️">
+Keyword search for the **hybrid** retriever is currently implemented only for the **pgvector** vector store. On other stores (FAISS, Qdrant, Milvus, etc.) the keyword half returns nothing, so `hybrid` quietly behaves like `classic` (vector-only).
+</Callout>
+
+Operators can restrict which retrievers are usable instance-wide with the `RETRIEVERS_ENABLED` setting; a per-source `retriever` value must be within that allow-list.
+
+### Exposure: prefetch vs. agentic tool
+
+`exposure` controls *how* a source's content is delivered to the model:
+
+- **`prefetch`** (default) — DocsGPT retrieves the top chunks up front and injects them into the prompt before the model answers. Best for focused Q&A over a source.
+- **`agentic_tool`** — The source is exposed to the model as a search tool it can call on demand, deciding when and what to look up (browse-as-you-go) rather than receiving a bulk prefetch. This is the default exposure for [Wiki sources](/Sources/Wiki-sources).
+
+### Pre-screening (LLM relevance filter)
+
+Pre-screening adds an optional map-reduce step between retrieval and answering: a base retriever fetches a wider set of candidates, an LLM screens them in batches, and only the most relevant survivors are passed to the answer. It improves precision on noisy sources at the cost of extra query-time LLM calls, so it is **off by default**.
+
+```json
+{
+  "retrieval": {
+    "chunks": 8,
+    "prescreen": {
+      "candidate_k": 40,
+      "batch_size": 10,
+      "max_keep": 8,
+      "model": null
+    }
+  }
+}
+```
+
+| Field | Default | Description |
+| --- | --- | --- |
+| `candidate_k` | `40` | Candidates fetched before screening. Must be `>= chunks`. |
+| `batch_size` | `10` | Candidates screened per LLM call. |
+| `max_keep` | `8` | Survivors kept after screening. Must be `<= candidate_k`. |
+| `model` | `null` | Model used for screening. `null` reuses the request's resolved model. |
+
+## Editing the config via API
+
+The config is edited with a `PATCH` to the source's config endpoint:
+
+```bash
+curl -X PATCH https://your-docsgpt/api/sources/<source_id>/config \
+  -H "Authorization: Bearer <token>" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "retrieval": { "retriever": "hybrid", "chunks": 4 },
+    "chunking":  { "strategy": "semantic" }
+  }'
+```
+
+The response echoes the stored config and a `requires_reingest` flag:
+
+```json
+{
+  "success": true,
+  "config": { "...": "..." },
+  "requires_reingest": true
+}
+```
+
+Notes:
+
+- Invalid values are rejected with `400` (strict validation on write).
+- The `kind` field (classic / wiki / graphrag) cannot be changed through this endpoint — converting a source to a [Wiki](/Sources/Wiki-sources) or enabling [GraphRAG](/Sources/GraphRAG) uses dedicated endpoints.
+- Editing requires ownership of the source or a team `editor` grant; viewers receive `403`.
+
+## Related
+
+- [GraphRAG](/Sources/GraphRAG) — knowledge-graph retrieval for a source.
+- [Wiki Sources](/Sources/Wiki-sources) — LLM-editable living documentation.
+- [Embeddings](/Models/embeddings) — the embedding model used during ingest and retrieval.

docs/content/Sources/Wiki-sources.mdx

@@ -0,0 +1,96 @@
+---
+title: Wiki Sources — Living, LLM-Editable Documentation
+description: Create a knowledge source that the agent can read and write — a living wiki it keeps up to date, with human edits, provenance stamps, and version safety.
+---
+
+import { Callout } from 'nextra/components'
+
+# Wiki Sources
+
+A **wiki source** is a knowledge source that the agent can both read *and* write. Instead of being a fixed set of ingested files, a wiki is a small set of Markdown pages that the LLM edits over time — recording what it learns, correcting stale information, and building living documentation. Humans can edit the same pages directly, and every change is stamped with who made it.
+
+Unlike a classic source, a wiki is **team-scoped, not per-user**: it is shared and edited at the source level, so a whole team works against the same living document.
+
+## How a wiki differs from a classic source
+
+| | Classic source | Wiki source |
+| --- | --- | --- |
+| Content | Ingested files, read-only | Markdown pages, read **and** write |
+| Who edits | You (re-upload to change) | The agent and humans |
+| Default exposure | `prefetch` (chunks injected up front) | `agentic_tool` (the agent browses pages on demand) |
+| Searchability | Embedded at ingest | Re-embedded automatically on every edit |
+| Scope | Per owner | Team-shareable, edited at source scope |
+
+Because a wiki defaults to the `agentic_tool` [exposure](/Sources/Per-source-configuration#exposure-prefetch-vs-agentic-tool), the agent navigates it as a tool — opening, searching, and editing pages as needed — rather than receiving a bulk prefetch.
+
+## How the agent edits a wiki
+
+The agent edits a wiki through an internal **Wiki tool** that is automatically scoped to one wiki source. It supports a small, edit-safe action surface:
+
+- **view** a page,
+- **create** or overwrite a page,
+- **str_replace** an exact, unique string,
+- **insert** at a line,
+- **delete** a page,
+- **rename** a page.
+
+Two safety properties matter:
+
+- **Provenance stamps.** Every page records whether its last change came from a human or the agent, so edits are traceable.
+- **Optimistic versioning.** Edits carry an expected version; if a page changed underneath, the edit is rejected rather than silently clobbering a concurrent change. String replacements must match exactly and uniquely.
+
+After any edit, the affected page is **re-embedded asynchronously** so the wiki stays searchable and the new content is immediately retrievable.
+
+<Callout type="info" emoji="ℹ️">
+Each wiki page is capped at 1 MB (1,000,000 bytes). Pages are addressed by path (the home page is `/index.md`).
+</Callout>
+
+## Creating a wiki
+
+In the UI, choose **New wiki source** when adding a source. From the API:
+
+```bash
+curl -X POST https://your-docsgpt/api/sources/wiki \
+  -H "Authorization: Bearer <token>" \
+  -H "Content-Type: application/json" \
+  -d '{ "name": "Team Handbook", "initial_content": "# Team Handbook\n\nWelcome." }'
+```
+
+- `name` (required) — the wiki source name.
+- `initial_content` (optional) — Markdown that seeds the home page `/index.md` and triggers its first re-embed.
+
+No ingestion task runs for a wiki; pages are authored directly. The response returns the new `source_id`.
+
+## Converting an existing source into a wiki
+
+You can turn an already-ingested source into a wiki so the agent can start maintaining it:
+
+```bash
+curl -X POST https://your-docsgpt/api/sources/<source_id>/wiki/convert \
+  -H "Authorization: Bearer <token>"
+```
+
+- A **blank** source is enabled inline (no task) and immediately becomes a wiki.
+- A source **with files** runs a conversion task that reassembles its existing chunks into wiki pages; poll the returned task for a per-file summary.
+- Conversion is rejected with `409` if the source is still ingesting — wait for it to finish first.
+
+<Callout type="warning" emoji="⚠️">
+Switching a source to (or from) wiki mode goes only through `POST /api/sources/<id>/wiki/convert`. It cannot be done through the [config PATCH endpoint](/Sources/Per-source-configuration#editing-the-config-via-api).
+</Callout>
+
+## Reading and editing pages directly
+
+Humans can read and edit wiki pages through the API (and the wiki viewer in the UI):
+
+```text
+GET  /api/sources/<source_id>/wiki/pages          # list pages
+GET  /api/sources/<source_id>/wiki/page?path=...  # fetch one page fresh
+PUT  /api/sources/<source_id>/wiki/page           # create or overwrite a page (human edit)
+```
+
+Human edits are stamped with `human` provenance and trigger the same re-embed as agent edits. Read access follows source sharing (owner or anyone the source is shared with); writing requires owner or team `editor` access.
+
+## Related
+
+- [Per-Source Configuration](/Sources/Per-source-configuration) — exposure and retrieval settings a wiki uses.
+- [Access Control & Teams](/Deploying/Access-Control) — sharing a wiki with a team.

docs/content/Sources/_meta.js

@@ -0,0 +1,14 @@
+export default {
+  "Per-source-configuration": {
+    "title": "🎛️ Per-Source Configuration",
+    "href": "/Sources/Per-source-configuration"
+  },
+  "GraphRAG": {
+    "title": "🕸️ GraphRAG",
+    "href": "/Sources/GraphRAG"
+  },
+  "Wiki-sources": {
+    "title": "📖 Wiki Sources",
+    "href": "/Sources/Wiki-sources"
+  }
+}

docs/content/Tools/_meta.js

@@ -0,0 +1,22 @@
+export default {
+  "basics": {
+    "title": "🔧 Tools Basics",
+    "href": "/Tools/basics"
+  },
+  "api-tool": {
+    "title": "🗝️ API Tool",
+    "href": "/Tools/api-tool"
+  },
+  "remote-device": {
+    "title": "🖥️ Remote Device",
+    "href": "/Tools/remote-device"
+  },
+  "artifacts-and-code-execution": {
+    "title": "📦 Artifacts and Code Execution",
+    "href": "/Tools/artifacts-and-code-execution"
+  },
+  "creating-a-tool": {
+    "title": "🛠️ Creating a Custom Tool",
+    "href": "/Tools/creating-a-tool"
+  }
+}

docs/content/Tools/api-tool.mdx

@@ -0,0 +1,153 @@
+---
+title: 🗝️ Generic API Tool
+description: Learn how to configure and use the API Tool in DocsGPT to connect with any RESTful API without writing custom code.
+---
+
+import { Callout } from 'nextra/components';
+import Image from 'next/image';
+
+# Using the Generic API Tool
+
+The API Tool provides a no-code/low-code solution to make DocsGPT interact with third-party or internal RESTful APIs. It acts as a bridge, allowing the Large Language Model (LLM) to leverage external services based on your chat interactions.
+ This guide will walk you through its capabilities, configuration, and best practices.
+
+## Introduction to the Generic API Tool
+
+**When to Use It:**
+    * Ideal for quickly integrating existing APIs where the interaction involves standard HTTP requests (GET, POST, PUT, DELETE).
+    * Suitable for fetching data to enrich answers (e.g., current weather, stock prices, product details).
+    * Useful for triggering simple actions in other systems (e.g., sending a notification, creating a basic task).
+
+**Contrast with Custom Python Tools:**
+    * **API Tool:** Best for straightforward API calls. Configuration is done through the DocsGPT UI.
+    * **Custom Python Tools:** Preferable when you need complex logic before or after the API call, handle non-standard authentication (like complex OAuth flows), manage multi-step API interactions, or require intricate data processing not easily managed by the LLM alone. See [Creating a Custom Tool](/Tools/creating-a-tool) for more.
+
+## Capabilities of the API Tool
+
+**Supported HTTP Methods:** You can configure actions using standard HTTP methods such as:
+    * `GET`: To retrieve data.
+    * `POST`: To submit data to create a new resource.
+    * `PUT`: To update an existing resource.
+    * `DELETE`: To remove a resource.
+
+**Request Configuration:**
+    * **Headers:** Define static or dynamic HTTP headers for authentication (e.g., API keys), content type specification, etc.
+    * **Query Parameters:** Specify URL query parameters, which can be static or dynamically filled by the LLM based on user input.
+    * **Request Body:** Define the structure of the request body (e.g., JSON), with fields that can be static or dynamically populated by the LLM.
+
+**Response Handling:**
+    * The API Tool executes the request and receives the raw response from the API (typically JSON or plain text).
+    * This raw response is then passed back to the LLM.
+    * The LLM uses this response, along with the context of your query and the description of the API tool action, to formulate an answer or decide on follow-up actions. The API tool itself doesn't deeply parse or transform the response beyond basic content type detection (e.g., loading JSON into a parsable object).
+
+## Configuring an API as a Tool
+
+You can configure the API Tool through the DocsGPT user interface, found in **Settings -> Tools**. When you add or modify an API Tool, you'll define specific actions that DocsGPT can perform.
+
+<Callout type="info">
+  The configuration involves defining how DocsGPT should call an API endpoint. Each configured API call essentially becomes a distinct "action" the LLM can choose to use.
+</Callout>
+
+Below is an example of how you might configure an API action, inspired by setting up a phone number validation service:
+
+<Image
+  src="/toolIcons/api-tool-example.png"
+  alt="API Tool configuration example for phone validation"
+  width={800}
+  height={450}
+  style={{ margin: '1em auto', display: 'block', borderRadius: '8px' }}
+/>
+_Figure 1: Example configuration for an API Tool action to validate phone numbers._
+
+**Defining an API Endpoint/Action:**
+
+When you configure a new API action, you'll fill in the following fields:
+
+-   **`Name`:** A user-friendly name for this specific API action (e.g., "Phone-check" as in the image, or more specific like "ValidateUSPhoneNumber"). This helps in managing your tools.
+-   **`Description`:** This is a **critical field**. Provide a clear and concise description of what the API action does, what kind of input it expects (implicitly), and what kind of output it provides. The LLM uses this description to understand when and how to use this action.
+-   **`URL`:** The full endpoint URL for the API request.
+-   **`HTTP Method`:** Select the appropriate HTTP method (e.g., GET, POST) from a dropdown.
+-   **`Headers`:** You can add custom HTTP headers as key-value pairs (Name, Value). Indicate if the value should be `Filled by LLM` or is static. If filled by LLM, provide a `Description` for the LLM.
+
+-   **`Query Parameters`:** For `GET` requests or when parameters are sent in the URL.
+    * **`Name`:** The name of the query parameter (e.g., `api_key`, `phone`).
+    * **`Type`:** The data type of the parameter (e.g., `string`).
+    * **`Filled by LLM` (Checkbox):**
+        - **Unchecked (Static):** The `Value` you provide will be used for every call (e.g., for an `api_key` that doesn't change).
+        - **Checked (Dynamic):** The LLM will extract the appropriate value from the user's chat query based on the `Description` you provide for this parameter. The `Value` field is typically left empty or contains a placeholder if `Filled by LLM` is checked.
+    * `Description`: Context for the LLM if the parameter is to be filled dynamically, or for your own reference if static.
+    * `Value`: The static value if not filled by LLM.
+
+-   **`Request Body`:** Used to send data (commonly JSON) to the API. Similar to Query Parameters, you define fields with `Name`, `Type`, whether it's `Filled by LLM`, a `Description` for dynamic fields, and a static `Value` if applicable.
+
+**Response Handling Guidance for the LLM:**
+
+While the API Tool configuration UI doesn't have explicit fields for defining response parsing rules (like JSONPath extractors), you significantly influence how the LLM handles the response through:
+    * **Tool Action `Description`:** Clearly state what kind of information the API returns (e.g., "This API returns a JSON object with 'status' and 'location' fields for the phone number."). This helps the LLM know what to look for in the API's output.
+    * **Prompt Engineering:** For more complex scenarios, you might need to adjust your global or agent-specific prompts to guide DocsGPT on how to interpret and present information from API tool responses. See [Customising Prompts](/Guides/Customising-prompts).
+
+## Using the Configured API Tool in Chat
+
+Once an API action is configured and enabled, DocsGPT's LLM can decide to use it based on your natural language queries.
+
+**Example (based on the phone validation tool in Figure 1):**
+
+1.  **User Query:** "Hey DocsGPT, can you check if +14155555555 is a valid phone number?"
+
+2.  **DocsGPT (LLM Orchestration):**
+    * The LLM analyzes the query.
+    * It matches the intent ("check if ... is a valid phone number") with the description of the "Phone-check" API action.
+    * It identifies `+14155555555` as the value for the `phone` parameter (which was marked as `Filled by LLM` with the description "Phone number to check").
+    * DocsGPT constructs the GET API request.
+3.  **API Tool Execution:**
+    * The API Tool makes the HTTP GET request.
+    * The external API (AbstractAPI) processes the request and returns a JSON response, e.g.:
+        ```json
+        {
+          "phone": "+14155555555",
+          "valid": true,
+          "format": {
+            "international": "+1 415-555-5555",
+            "national": "(415) 555-5555"
+          },
+          "country": {
+            "code": "US",
+            "name": "United States",
+            "prefix": "+1"
+          },
+          "location": "California",
+          "type": "Landline"
+        }
+        ```
+
+4.  **DocsGPT Response Formulation:**
+    * The API Tool passes this JSON response back to the LLM.
+    * The LLM, guided by the tool's description and the user's original query, extracts relevant information and formulates a user-friendly answer.
+    * **DocsGPT Chat Response:** "Yes, +14155555555 appears to be a valid landline phone number in California, United States."
+
+## Advanced Tips and Best Practices
+
+**Clear Description is the Key:** The LLM relies heavily on the `Description` field of the API action and its parameters. Make them unambiguous and action-oriented. Clearly state what the tool does and what kind of input it expects (even if implicitly through parameter descriptions). 
+
+**Iterative Testing:** After configuring an API tool, test it with various phrasings of user queries to ensure the LLM triggers it correctly and interprets the response as expected.
+  
+**Error Handling:**
+    * If an API call fails, the API Tool will return an error message and status code from the `requests` library or the API itself. The LLM may relay this error or try to explain it.
+    * Check DocsGPT's backend logs for more detailed error information if you encounter issues.
+  
+**Security Considerations:**
+    * **API Keys:** Be mindful of API keys and other sensitive credentials. The example image shows an API key directly in the configuration. For production or shared environments avoid exposing configurations with sensitive keys.
+    * **Rate Limits:** Be aware of the rate limits of the APIs you are integrating. Frequent calls from DocsGPT could exceed these limits.
+    * **Data Privacy:** Consider the data privacy implications of sending user query data to third-party APIs.
+-   **Idempotency:** For tools that modify data (POST, PUT, DELETE), be aware of whether the API operations are idempotent to avoid unintended consequences from repeated calls if the LLM retries an action.
+
+## Limitations
+
+While powerful, the Generic API Tool has some limitations:
+
+-   **Complex Authentication:** Advanced authentication flows like OAuth 2.0 (especially 3-legged OAuth requiring user redirection) or custom signature-based authentication often require custom Python tools.
+-   **Multi-Step API Interactions:** If a task requires multiple API calls that depend on each other (e.g., fetch a list, then for each item, fetch details), this kind of complex chaining and logic is better handled by a custom Python tool.
+-   **Complex Data Transformations:** If the API response needs significant transformation or processing before being useful to the LLM, a custom Python tool offers more flexibility.
+-   **Real-time Streaming (SSE, WebSockets):** The tool is designed for request-response interactions, not for maintaining persistent streaming connections.
+
+For scenarios that exceed these limitations, developing a [Custom Python Tool](/Tools/creating-a-tool) is the recommended approach.

docs/content/Tools/artifacts-and-code-execution.mdx

@@ -0,0 +1,101 @@
+---
+title: 📦 Artifacts and Code Execution
+description: Generate editable documents, run sandboxed code, and read files with the DocsGPT artifact, code executor, and read document tools.
+---
+
+import { Callout } from 'nextra/components';
+
+# Artifacts and Code Execution
+
+DocsGPT can generate documents, run code, and read files for you during a chat or inside an agent workflow. Three built-in tools work together for this:
+
+- **Artifact** builds and edits documents such as slide decks, Word documents, spreadsheets, PDFs, and HTML.
+- **Code Executor** runs code in a sandboxed session and turns any files the code writes into downloadable artifacts.
+- **Read Document** parses an uploaded or produced file (PDF, Word, PowerPoint, and more) into text, markdown, or structured data.
+
+All three are opt-in: enable Artifact and Code Executor per agent in the tool picker (they need a running [sandbox runner](https://github.com/arc53/DocsGPT/tree/main/deployment/sandbox)), and Read Document is available inside workflows. A new agent starts with no tools enabled.
+
+## Artifacts
+
+An **artifact** is a file the assistant produces that you can open, download, and edit later. Artifacts are versioned: every edit appends a new version at the same artifact, so you can review or restore an earlier one. Bytes are stored on the server and are never passed through the model, so large files stay fast and cheap to work with.
+
+In a normal chat, a produced artifact appears as a chip under the assistant's reply (for example, "Artifact"). Open it to:
+
+- preview the content in a side panel (HTML and similar kinds render inline),
+- download the file,
+- ask for changes, which create a new version.
+
+### The Artifact tool
+
+The Artifact tool keeps a structured spec as the source of truth and renders the file from it. Editing means changing the spec and re-rendering, which keeps a clean and diffable version history. It exposes three actions:
+
+- **create** a new artifact from a kind and a spec,
+- **edit** an existing artifact with a targeted change (preferred for small edits),
+- **rewrite** an artifact with a full replacement spec.
+
+Supported kinds are presentation (.pptx), document (.docx), spreadsheet (.xlsx), pdf, and html.
+
+## Code Executor
+
+The Code Executor runs Python in a sandboxed, stateful session bound to your conversation (or to a workflow run). Files the code writes into the workspace are captured as artifacts automatically, so a script that produces `report.csv` gives you a downloadable artifact with no extra steps.
+
+Key points:
+
+- The session is stateful, so variables, imports, and files persist between calls in the same conversation while it is kept alive.
+- Only a compact summary is returned to the model (an output tail plus artifact references), never raw file bytes.
+- Each run has a fixed wall-clock limit of about 60 seconds. For longer work, start it in the background and check back with more calls.
+- Install packages from inside the code itself when you need them.
+
+<Callout type="warning">
+  By default the Code Executor runs model-authored code without an approval
+  prompt. Turn on per-action approval on the agent if you want a human to
+  confirm each run.
+</Callout>
+
+### Run inputs
+
+You can pass files into a run. Each input accepts a short reference returned by a previous artifact action, a full artifact id, or the name or id of a file you attached to the conversation. The referenced files are placed in the workspace before the code runs.
+
+## Read Document
+
+Read Document parses a file into text, markdown, structured JSON (with tables), or chunks. It is meant for workflows where a node needs the contents of an uploaded or produced document, for example a compliance flow that extracts fields and validates them against a JSON schema.
+
+Parsing runs in the DocsGPT parsing worker rather than the sandbox, so it uses the same document engine as the rest of the product and works with every sandbox backend.
+
+## Using these tools in workflows
+
+Inside an agent workflow the same building blocks are available:
+
+- A **Code node** runs a script bound to the run, with the workflow state available to it as data.
+- Agent and code nodes can receive selected documents as inputs and hand files to each other by reference.
+- Produced artifacts are visible from the workflow run view.
+
+See [Workflow Nodes](/Agents/nodes) for the node-level details.
+
+## Configuration
+
+Code execution and rendering run inside a sandbox. DocsGPT ships two backends, selected with `SANDBOX_BACKEND`:
+
+- **`jupyter`** (default): a self-hosted runner that you operate. See the runner setup notes under `deployment/sandbox`.
+- **`daytona`**: Daytona Cloud, a managed per-session sandbox. Set `DAYTONA_API_KEY` and choose a region with `DAYTONA_TARGET`.
+
+### Render libraries on Daytona
+
+Artifact rendering imports `python-pptx`, `python-docx`, `openpyxl`, and `reportlab` inside the sandbox. The self-hosted runner already has them. Daytona's default image does not, so presentation, document, spreadsheet, and pdf rendering fail there until you point Daytona at an image that includes them. HTML and markdown artifacts need no extra libraries and work on any image.
+
+Build a snapshot with the libraries once, then set `DAYTONA_SNAPSHOT` to it:
+
+```bash
+# Reads DAYTONA_API_KEY / DAYTONA_API_URL / DAYTONA_TARGET from your environment:
+python scripts/build_daytona_snapshot.py
+# then set in your environment:
+#   DAYTONA_SNAPSHOT=docsgpt-artifacts-py312
+```
+
+### Useful settings
+
+- `SANDBOX_BACKEND`: `jupyter` or `daytona`.
+- `SANDBOX_EXEC_TIMEOUT`: per-run wall-clock cap in seconds.
+- `SANDBOX_MAX_TTL`: upper bound on how long a kept-alive session lives.
+- `DAYTONA_API_KEY`, `DAYTONA_TARGET`, `DAYTONA_SNAPSHOT`: Daytona Cloud settings.
+- Per-user artifact quotas bound how many artifacts and how much storage each user can hold.

docs/content/Tools/basics.mdx

@@ -0,0 +1,136 @@
+---
+title: Tools Basics - Enhancing DocsGPT Capabilities
+description: Understand what DocsGPT Tools are, how they work, and explore the built-in tools available to extend DocsGPT's functionality.
+---
+
+import { Callout } from 'nextra/components';
+import Image from 'next/image';
+import { ToolCards } from '../../components/ToolCards';
+
+# Understanding DocsGPT Tools
+
+DocsGPT Tools are powerful extensions that significantly enhance the capabilities of your DocsGPT application. 
+They allow DocsGPT to move beyond its core function of retrieving information from your documents and enable it to perform actions, 
+interact with external data sources, and integrate with other services. You can find and configure available tools within 
+the "Tools" section of the DocsGPT application settings in the user interface.
+
+## What are Tools?
+
+-   **Purpose:** The primary purpose of Tools is to bridge the gap between understanding a user's request (natural language processing by the LLM) and executing a tangible action. This could involve fetching live data from the web, sending notifications, running code snippets, querying databases, or interacting with third-party APIs.
+
+-   **LLM as an Orchestrator:** The Large Language Model (LLM) at the heart of DocsGPT is designed to act as an intelligent orchestrator. Based on your query and the declared capabilities of the available tools (defined in their metadata), the LLM decides if a tool is needed, which tool to use, and what parameters to pass to it.
+
+-   **Action-Oriented Interactions:** Tools enable more dynamic and action-oriented interactions. For example:
+    * *"What's the latest news on renewable energy?"* - This might trigger a web search tool to fetch current articles.
+    * *"Fetch the order status for customer ID 12345 from our database."* - This could use a database tool.
+    * *"Summarize the content of this webpage and send the summary to the #general channel on Telegram."* - This might involve a web scraping tool followed by a Telegram notification tool.
+
+## Overview of Built-in Tools
+
+DocsGPT includes a suite of pre-built tools designed to expand its capabilities out-of-the-box. Below is an overview of the currently available tools. 
+
+<ToolCards
+  items={[
+    {
+      title: 'API Tool',
+      link: '/Tools/api-tool',
+      description: 'A highly flexible tool that allows DocsGPT to interact with virtually any API without needing to write custom Python code.'
+    },
+    {
+      title: 'Brave Search',
+      link: 'https://github.com/arc53/DocsGPT/blob/main/application/agents/tools/brave.py',
+      description: 'Enables DocsGPT to perform real-time web and image searches using the Brave Search API. Requires an API key.'
+    },
+    {
+      title: 'DuckDuckGo Search',
+      link: 'https://github.com/arc53/DocsGPT/blob/main/application/agents/tools/duckduckgo.py',
+      description: 'Performs web and image searches using DuckDuckGo. No API key required.'
+    },
+    {
+      title: 'CryptoPrice',
+      link: 'https://github.com/arc53/DocsGPT/blob/main/application/agents/tools/cryptoprice.py',
+      description: 'Fetches the current price of specified cryptocurrencies using the CryptoCompare public API.'
+    },
+    {
+      title: 'Ntfy',
+      link: 'https://github.com/arc53/DocsGPT/blob/main/application/agents/tools/ntfy.py',
+      description: 'Allows DocsGPT to send push notifications to ntfy topics on a specified server, ideal for alerts and updates.'
+    },
+    {
+      title: 'Telegram Bot',
+      link: 'https://github.com/arc53/DocsGPT/blob/main/application/agents/tools/telegram.py',
+      description: 'Allows DocsGPT to send messages or images to Telegram chats via a Telegram Bot. Requires a bot token and chat ID.'
+    },
+    {
+      title: 'PostgreSQL Database',
+      link: 'https://github.com/arc53/DocsGPT/blob/main/application/agents/tools/postgres.py',
+      description: 'Connects to a PostgreSQL database to execute SQL queries and retrieve schema information.'
+    },
+    {
+      title: 'Read Webpage (browser)',
+      link: 'https://github.com/arc53/DocsGPT/blob/main/application/agents/tools/read_webpage.py',
+      description: 'Fetches the HTML content of a URL and converts it to Markdown for the agent to read.'
+    },
+    {
+      title: 'Remote Device',
+      link: '/Tools/remote-device',
+      description: 'Runs shell commands on a paired remote machine through the docsgpt-cli host. See the Remote Device guide.'
+    },
+    {
+      title: 'MCP Tool',
+      link: '/Guides/Integrations/mcp-tool-integration',
+      description: 'Connects to remote Model Context Protocol (MCP) servers to access their dynamic tools and resources.'
+    },
+    {
+      title: 'Memory',
+      link: 'https://github.com/arc53/DocsGPT/blob/main/application/agents/tools/memory.py',
+      description: 'Stores and retrieves information across conversations through a per-user memory file directory.'
+    },
+    {
+      title: 'Notepad',
+      link: 'https://github.com/arc53/DocsGPT/blob/main/application/agents/tools/notes.py',
+      description: 'A single editable note. Supports viewing, overwriting, and string replacement.'
+    },
+    {
+      title: 'Todo List',
+      link: 'https://github.com/arc53/DocsGPT/blob/main/application/agents/tools/todo_list.py',
+      description: 'Manages todo items — creating, viewing, updating, and deleting todos.'
+    }
+  ]}
+/>
+
+## Default Chat Tools
+
+In a regular chat (no custom agent), DocsGPT can enable a small set of tools automatically so the assistant is useful out of the box. These are the **default chat tools**, controlled by the `DEFAULT_CHAT_TOOLS` setting:
+
+```env
+DEFAULT_CHAT_TOOLS=memory,read_webpage,scheduler
+```
+
+- Default tools are config-free and run with synthetic, deterministic tool IDs (no manual setup needed).
+- Each user can opt out of individual default tools from their settings; the disabled list is stored per user.
+- Some default tools are excluded from **headless runs** (scheduled tasks and webhook triggers). For example, `scheduler` is skipped in those runs to prevent a scheduled task from chaining new schedules on every fire.
+
+To change the defaults for the whole instance, set `DEFAULT_CHAT_TOOLS` to a comma-separated list of tool names. See [App Configuration](/Deploying/DocsGPT-Settings) for the full settings reference.
+
+## Using Tools in DocsGPT (User Perspective)
+
+Interacting with tools in DocsGPT is designed to be intuitive:
+
+1.  **Natural Language Interaction:** As a user, you typically interact with DocsGPT using natural language queries or commands. The LLM within DocsGPT analyzes your input to determine if a specific task can or should be handled by one of the available and configured tools.
+
+2.  **Configuration in UI:**
+    * Tools are generally managed and configured within the DocsGPT application's settings, found under a "Tools" section in the GUI.
+    * For tools that interact with external services (like Brave Search, Telegram, or any service via the API Tool), you might need to provide authentication credentials (e.g., API keys, tokens) or specific endpoint information during the tool's setup in the UI.
+
+3.  **Prompt Engineering for Tools:** While the LLM aims to intelligently use tools, for more complex or reliable agent-like behaviors, you might need to customize the system prompts. Modifying the prompt can guide the LLM on when and how to prioritize or chain tools to achieve specific outcomes, especially if you're building an agent designed to perform a certain sequence of actions every time. For more on this, see [Customising Prompts](/Guides/Customising-prompts).
+
+## Advancing with Tools
+
+Understanding the basics of DocsGPT Tools opens up many possibilities:
+
+* **Leverage the API Tool:** For quick integrations with numerous external services, explore the [API Tool Detailed Guide](/Tools/api-tool).
+* **Develop Custom Tools:** If you have specific needs not covered by built-in tools or the generic API tool, you can develop your own. See our guide on `[Developing Custom Tools](/Tools/creating-a-tool)` (placeholder for now).
+* **Build AI Agents:** Tools are the fundamental building blocks for creating sophisticated AI agents within DocsGPT. Explore how these can be combined by looking into the `[Agents section/tab concept - link to be added once available]`.
+
+By harnessing the power of Tools, you can transform DocsGPT into a more versatile and proactive assistant tailored to your unique workflows.

docs/content/Tools/creating-a-tool.mdx

@@ -0,0 +1,186 @@
+---
+title: 🛠️ Creating a Custom Tool
+description: Learn how to create custom Python tools to extend DocsGPT's functionality and integrate with various services or perform specific actions.
+---
+
+import { Callout } from 'nextra/components';
+import { Steps } from 'nextra/components';
+
+# 🛠️ Creating a Custom Python Tool
+
+This guide provides developers with a comprehensive, step-by-step approach to creating their own custom tools for DocsGPT. By developing custom tools, you can significantly extend DocsGPT's capabilities, enabling it to interact with new data sources, services, and perform specialized actions tailored to your unique needs.
+
+## Introduction to Custom Tool Development
+
+### Why Create Custom Tools?
+
+While DocsGPT offers a range of built-in tools and a versatile API Tool, there are many scenarios where a custom Python tool is the best solution:
+
+* **Integrating with Proprietary Systems:** Connect to internal APIs, databases, or services that are not publicly accessible or require complex authentication.
+* **Adding Domain-Specific Functionalities:** Implement logic specific to your industry or use case that isn't covered by general-purpose tools.
+* **Automating Unique Workflows:** Create tools that orchestrate multiple steps or interact with systems in a way unique to your operational needs.
+* **Connecting to Any System with an Accessible Interface:** If you can interact with a system programmatically using Python (e.g., through libraries, SDKs, or direct HTTP requests), you can likely build a DocsGPT tool for it.
+* **Complex Logic or Data Transformation:** When API interactions require intricate logic before sending a request or after receiving a response, or when data needs significant transformation that is difficult for an LLM to handle directly.
+
+### Prerequisites
+
+Before you begin, ensure you have:
+
+* A solid understanding of Python programming.
+* Familiarity with the DocsGPT project structure, particularly the `application/agents/tools/` directory where custom tools reside.
+* Basic knowledge of how APIs work, as many tools involve interacting with external or internal APIs.
+* Your DocsGPT development environment set up. If not, please refer to the [Setting Up a Development Environment](/Deploying/Development-Environment) guide.
+
+## The Anatomy of a DocsGPT Tool
+
+Custom tools in DocsGPT are Python classes that inherit from a base `Tool` class and implement specific methods to define their behavior, capabilities, and configuration needs.
+
+The **foundation** for all custom tools is the abstract base class, located in `application/agents/tools/base.py`. Your custom tool class **must** inherit from this class.
+
+### Essential Methods to Implement
+
+Your custom tool class needs to implement the following methods:
+
+1. **`__init__(self, config: dict)`**
+    
+    - **Purpose:** The constructor for your tool. It's called when DocsGPT initializes the tool.
+    - **Usage:** This method is typically used to receive and store tool-specific configurations passed via the `config` dictionary. This dictionary is populated based on the tool's settings, often configured through the DocsGPT UI or environment variables. For example, you would store API keys, base URLs, or database connection strings here.
+    - **Example** (`brave.py`)**:**
+        ``` python
+        class BraveSearchTool(Tool):
+            def __init__(self, config):
+                self.config = config
+                self.token = config.get("token", "") # API Key for Brave Search
+                self.base_url = "https://api.search.brave.com/res/v1"
+        ```
+
+2. **`execute_action(self, action_name: str, **kwargs) -> dict`**
+    
+    - **Purpose:** This is the workhorse of your tool. The LLM, acting as an agent, calls this method when it decides to use one of the actions your tool provides.
+    - **Parameters:**
+        - `action_name` (str): A string specifying which of the tool's actions to run (e.g., "brave_web_search").
+        - `**kwargs` (dict): A dictionary containing the parameters for that specific action. These parameters are defined in the tool's metadata (`get_actions_metadata()`) and are extracted or inferred by the LLM from the user's query.
+    - **Return Value:** A dictionary containing the result of the action. It's good practice to include keys like:
+        - `status_code` (int): An HTTP-like status code (e.g., 200 for success, 500 for error).
+        - `message` (str): A human-readable message describing the outcome.
+        - `data` (any): The actual data payload returned by the action (if applicable).
+        - `error` (str): An error message if the action failed.
+    - **Example (`read_webpage.py`):**
+        
+        ``` python
+        def execute_action(self, action_name: str, **kwargs) -> str:
+            if action_name != "read_webpage":
+                return f"Error: Unknown action '{action_name}'. This tool only supports 'read_webpage'."
+        
+            url = kwargs.get("url")
+            if not url:
+                return "Error: URL parameter is missing."
+            # ... (logic to fetch and parse webpage) ...
+            try:
+                # ...
+                return markdown_content 
+            except Exception as e:
+                return f"Error processing URL {url}: {e}"
+        ```
+        
+        A more structured return:
+        
+        ``` python
+        # ... inside execute_action
+        try:
+            # ... logic ...
+            return {"status_code": 200, "message": "Webpage read successfully", "data": markdown_content}
+        except Exception as e:
+            return {"status_code": 500, "message": f"Error processing URL {url}", "error": str(e)}
+        ```
+        
+3. **`get_actions_metadata(self) -> list`**
+    
+    - **Purpose:** This method is **critical** for the LLM to understand what your tool can do, when to use it, and what parameters it needs. It effectively advertises your tool's capabilities.
+    - **Return Value:** A list of dictionaries. Each dictionary describes one distinct action the tool can perform and must follow a specific JSON schema structure.
+        - `name` (str): A unique and descriptive name for the action (e.g., `mytool_get_user_details`). It's a common convention to prefix with the tool name to avoid collisions.
+        - `description` (str): A clear, concise, and unambiguous description of what the action does. **Write this for the LLM.** The LLM uses this description to decide if this action is appropriate for a given user query.
+        - `parameters` (dict): A JSON Schema object defining the parameters that the action expects. This schema tells the LLM what arguments are needed, their types, and which are required.
+            - `type`: Should always be `"object"`.
+            - `properties`: A dictionary where each key is a parameter name, and the value is an object defining its `type` (e.g., "string", "integer", "boolean") and `description`.
+            - `required`: A list of strings, where each string is the name of a parameter that is mandatory for the action.
+    - **Example (`postgres.py` - partial):**
+        
+        ``` python
+        def get_actions_metadata(self):
+            return [
+                {
+                    "name": "postgres_execute_sql",
+                    "description": "Execute an SQL query against the PostgreSQL database...",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "sql_query": {
+                                "type": "string",
+                                "description": "The SQL query to execute.",
+                            },
+                        },
+                        "required": ["sql_query"],
+                        "additionalProperties": False, # Good practice to prevent unexpected params
+                    },
+                },
+                # ... other actions like postgres_get_schema
+            ]
+        ```
+        
+4. **`get_config_requirements(self) -> dict`**
+    
+    - **Purpose:** Defines the configuration parameters that your tool needs to function (e.g., API keys, specific base URLs, connection strings, default settings). This information can be used by the DocsGPT UI to dynamically render configuration fields for your tool or for validation.
+    - **Return Value:** A dictionary where keys are the configuration item names (which will be keys in the `config` dict passed to `__init__`) and values are dictionaries describing each requirement:
+        - `type` (str): The expected data type of the config value (e.g., "string", "boolean", "integer").
+        - `description` (str): A human-readable description of what this configuration item is for.
+        - `secret` (bool, optional): Set to `True` if the value is sensitive (e.g., an API key) and should be masked or handled specially in UIs. Defaults to `False`.
+    - **Example (`brave.py`):**
+        
+        ``` python
+        def get_config_requirements(self):
+            return {
+                "token": { # This 'token' will be a key in the config dict for __init__
+                    "type": "string",
+                    "description": "Brave Search API key for authentication",
+                    "secret": True
+                },
+            }
+        ```    
+
+## Tool Registration and Discovery
+
+DocsGPT's ToolManager (located in application/agents/tools/tool_manager.py) automatically discovers and loads tools.
+
+As long as your custom tool:
+
+1. Is placed in a Python file within the `application/agents/tools/` directory (and the filename is not `base.py` or starts with `__`).
+2. Correctly inherits from the `Tool` base class.
+3. Implements all the abstract methods (`execute_action`, `get_actions_metadata`, `get_config_requirements`).
+
+The `ToolManager` should be able to load it when DocsGPT starts. 
+
+## Configuration & Secrets Management
+
+- **Configuration Source:** The `config` dictionary passed to your tool's `__init__` method is typically populated from settings defined in the DocsGPT UI (if available for the tool) or from environment variables/configuration files that DocsGPT loads (see [⚙️ App Configuration](/Deploying/DocsGPT-Settings)). The keys in this dictionary should match the names you define in `get_config_requirements()`.
+- **Secrets:** Never hardcode secrets (like API keys or passwords) directly into your tool's Python code. Instead, define them as configuration requirements (using `secret: True` in `get_config_requirements()`) and let DocsGPT's configuration system inject them via the `config` dictionary at runtime. This ensures that secrets are managed securely and are not exposed in your codebase.
+
+## Best Practices for Tool Development
+
+- **Atomicity:** Design tool actions to be as atomic (single, well-defined purpose) as possible. This makes them easier for the LLM to understand and combine.
+- **Clarity in Metadata:** Ensure action names and descriptions in `get_actions_metadata()` are extremely clear, specific, and unambiguous. This is the primary way the LLM understands your tool.
+- **Robust Error Handling:** Implement comprehensive error handling within your `execute_action` logic (and the private methods it calls). Return informative error messages in the result dictionary so the LLM or user can understand what went wrong.
+- **Security:**
+    - Be mindful of the security implications of your tool, especially if it interacts with sensitive systems or can execute arbitrary code/queries.
+    - Validate and sanitize any inputs, especially if they are used to construct database queries or shell commands, to prevent injection attacks.
+- **Performance:** Consider the performance implications of your tool's actions. If an action is slow, it will impact the user experience. Optimize where possible.
+
+## (Optional) Contributing Your Tool
+
+If you develop a custom tool that you believe could be valuable to the broader DocsGPT community and is general-purpose:
+
+1. Ensure it's well-documented (both in code and with clear metadata).
+2. Make sure it adheres to the best practices outlined above.
+3. Consider opening a Pull Request to the [DocsGPT GitHub repository](https://github.com/arc53/DocsGPT) with your new tool, including any necessary documentation updates.
+
+By following this guide, you can create powerful custom tools that extend DocsGPT's capabilities to your specific operational environment.

docs/content/Tools/remote-device.mdx

@@ -0,0 +1,111 @@
+---
+title: 🖥️ Remote Device
+description: Run shell commands on a paired remote machine from a DocsGPT agent using docsgpt-cli host.
+---
+
+import { Callout } from 'nextra/components';
+
+# Remote Device Tool
+
+The Remote Device tool lets a DocsGPT agent run shell commands on a machine you control, such as a server, a Raspberry Pi, or your own laptop. You install `docsgpt-cli` on that machine, run it in `host` mode, and pair it with your DocsGPT account. The paired device then shows up as a tool you can attach to any agent.
+
+The machine connects outward to DocsGPT, so it works behind NAT or a firewall without opening any inbound ports.
+
+## How it works
+
+1. You run `docsgpt-cli host` on the target machine. It pairs to your account and keeps a lightweight connection open: it polls while idle and streams while a command is running.
+2. When an agent calls the device's `run_command` action, DocsGPT sends the command down to the daemon.
+3. The daemon runs the command locally, streams stdout and stderr back, and the agent uses the output to continue.
+4. Every invocation is recorded in the device's activity log.
+
+## Prerequisites
+
+- `docsgpt-cli` installed on the machine you want to control. See the [installation instructions](https://github.com/arc53/DocsGPT-cli#installation): download a binary from the [releases page](https://github.com/arc53/DocsGPT-cli/releases), use Homebrew (`brew tap arc53/docsgpt-cli && brew install docsgpt-cli`), or build from source.
+- Outbound internet access from that machine to your DocsGPT instance.
+
+<Callout type="info">
+  The `host` commands require a docsgpt-cli build with remote-device support. If `docsgpt-cli host` is not recognized, update to the latest release or build from source.
+</Callout>
+
+## Pair a device
+
+Pairing uses a short one-time code (a device-code style flow).
+
+1. In DocsGPT, go to **Settings -> Tools**, click **Add tool**, and choose **Remote Device**.
+2. Give it a name, an optional description (the agent sees this, so describe what the machine is for), and pick an approval mode. DocsGPT then shows a pairing code such as `ABCD-WXYZ` and the command to run.
+3. On the target machine, run:
+   ```bash
+   docsgpt-cli host pair --url https://your-docsgpt-instance
+   ```
+   Enter the code when prompted. (Omit `--url` to use the default cloud instance.)
+4. The UI switches to "Paired" once the code is redeemed. If you ran the command at a terminal, the CLI then offers to start the daemon or install it as a service.
+
+## Run the daemon
+
+Start it in the foreground:
+
+```bash
+docsgpt-cli host
+```
+
+You will see a startup banner, then periodic "idle" heartbeats. Press Ctrl-C to stop.
+
+To keep it running across reboots, install it as a service:
+
+```bash
+# Linux (systemd). As root this installs a system service; otherwise a user service.
+docsgpt-cli host install-service
+
+# macOS (launchd). The default is a per-user LaunchAgent that starts on login.
+docsgpt-cli host install-service
+
+# Always-on machine that starts at boot (Linux example):
+sudo docsgpt-cli host install-service --system --user $USER
+```
+
+Remove the service with `docsgpt-cli host uninstall-service`.
+
+## Approval modes
+
+The approval mode is set per device, either in the pairing form or later on the device's page under **Settings -> Tools**. There are two modes:
+
+- **Ask** (default): every command pauses for your approval before it runs. You approve it from the chat. You can also choose "approve and don't ask again" to auto-approve that command pattern in the future.
+- **Full access**: commands run without asking.
+
+<Callout type="warning">
+  A built-in safety denylist always applies, even in Full access mode. Catastrophic commands (for example `rm -rf /`, fork bombs, writing directly to a disk device, or `git push --force`) still pause for explicit approval and cannot be bypassed.
+</Callout>
+
+Compound commands are split on operators such as `&&`, `||`, `;`, and `|`, and each part is checked on its own, so a dangerous part cannot be hidden behind a safe one.
+
+## Attach to an agent
+
+A paired device behaves like any other tool. Open or create an agent, add the device from the tool picker, and the agent can call its `run_command` action. The picker shows whether the device is currently online.
+
+If you attach more than one device to an agent, give each a clear description so the agent can pick the right one.
+
+## Manage a device
+
+Open the device from **Settings -> Tools** to:
+
+- See its status (online or offline, with last-seen time), host, OS, and CLI version.
+- Change its name, description, or approval mode.
+- View recent activity (the command audit log).
+- Revoke it.
+
+From the CLI:
+
+```bash
+docsgpt-cli host status   # live status from the server
+docsgpt-cli host revoke   # revoke on the server and clear local state
+docsgpt-cli host reset    # clear local pairing only (leaves the server-side device)
+```
+
+Revoking a device stops its daemon: the next time it checks in it sees the revocation, prints a message, and exits. Under a service manager it will not be restarted.
+
+## Security notes
+
+- The machine connects outward only. No inbound ports are opened.
+- Each device has its own token, stored hashed on the server and revocable at any time.
+- Prefer **Ask** mode for any machine with sensitive data on it. Use **Full access** only on machines you are comfortable letting an agent drive unattended, and remember that the denylist is the only automatic guard in that mode.
+- Every command is logged on the server and visible in the device's activity log.

docs/content/_meta.js

@@ -0,0 +1,20 @@
+export default {
+  "index": "Home",
+  "quickstart": "Quickstart",
+  "upgrading": "Upgrading",
+  "Deploying": "Deploying",
+  "Models": "Models",
+  "Sources": "Sources",
+  "Tools": "Tools",
+  "Agents": "Agents",
+  "Extensions": "Extensions",
+  "https://gptcloud.arc53.com/": {
+    "title": "API",
+    "href": "https://gptcloud.arc53.com/"
+  },
+  "Guides": "Guides",
+  "changelog": {
+    "title": "Changelog",
+    "display": "hidden"
+  }
+}

docs/content/changelog.mdx

@@ -0,0 +1,3 @@
+---
+title: 'Changelog'
+---

docs/content/index.mdx

@@ -0,0 +1,95 @@
+---
+title: 'Home'
+description: Documentation of DocsGPT - quickstart, deployment guides, model configuration, and widget integration documentation.
+---
+import { Cards } from 'nextra/components'
+import Image from 'next/image'
+
+export const allGuides = {
+  "quickstart": {
+    "title": "⚡️ Quickstart",
+    "href": "/quickstart"
+  },
+  "DocsGPT-Settings": {
+    "title": "⚙️ App Configuration",
+    "href": "/Deploying/DocsGPT-Settings"
+  },
+  "Docker-Deploying": {
+    "title": "🛳️ Docker Setup",
+    "href": "/Deploying/Docker-Deploying"
+  },
+  "Development-Environment": {
+    "title": "🛠️Development Environment",
+    "href": "/Deploying/Development-Environment"
+  },
+  "https://gptcloud.arc53.com/": {
+    "title": "🧑‍💻️ API",
+    "href": "https://gptcloud.arc53.com/",
+    "newWindow": true
+  },
+  "cloud-providers": {
+    "title": "☁️ Cloud Providers",
+    "href": "/Models/cloud-providers"
+  },
+  "local-inference": {
+    "title": "🖥️ Local Inference",
+    "href": "/Models/local-inference"
+  },
+  "embeddings": {
+    "title": "📝 Embeddings",
+    "href": "/Models/embeddings"
+  },
+  "api-key-guide": {
+    "title": "🔑 Getting API key",
+    "href": "/Extensions/api-key-guide"
+  },
+  "chat-widget": {
+    "title": "💬️ Chat Widget",
+    "href": "/Extensions/chat-widget"
+  },
+  "search-widget": {
+    "title": "🔎 Search Widget",
+    "href": "/Extensions/search-widget"
+  },
+  "Customising-prompts": {
+    "title": "️💻 Customising Prompts",
+    "href": "/Guides/Customising-prompts"
+  }
+};
+
+#  **DocsGPT 🦖**
+
+DocsGPT is an open-source genAI tool that helps users get reliable answers from any knowledge source, while avoiding hallucinations. It enables quick and reliable information retrieval, with tooling and agentic system capability built in, including speech-to-text workflows for chat and audio knowledge ingestion.
+
+
+<video controls width={1920} height={1080} muted autoPlay loop playsInline>
+  <source src="https://d3dg1063dc54p9.cloudfront.net/videos/demov4.mp4" type="video/mp4" />
+  Your browser does not support the video tag.
+</video>
+
+
+Try it yourself: [https://www.docsgpt.cloud/](https://www.docsgpt.cloud/)
+
+### Features:
+- **🗂️ Wide Format Support:** Reads PDF, DOCX, CSV, XLSX, EPUB, MD, RST, HTML, MDX, JSON, PPTX, images, and audio files such as MP3, WAV, M4A, OGG, and WebM.
+- **🎙️ Speech Workflows:** Record voice input into chat, transcribe on the backend, and index uploaded audio files as searchable source material.
+- **🌐 Web & Data Integration:** Ingests from URLs, sitemaps, Reddit, GitHub and web crawlers.
+- **✅ Reliable Answers:** Get accurate, hallucination-free responses with source citations viewable in a clean UI.
+- **🔑 Streamlined API Keys:**  Generate keys linked to your settings, documents, and models, simplifying chatbot and integration setup.
+- **🔗 Actionable Tooling:** Connect to APIs, tools, and other services to enable LLM actions.
+- **🧩 Pre-built Integrations:** Use readily available HTML/React chat widgets, search tools, Discord/Telegram bots, and more.
+- **🔌 Flexible Deployment:** Works with major LLMs (OpenAI, Google, Anthropic) and local models (Ollama, llama_cpp).
+- **🏢 Secure & Scalable:** Run privately and securely with Kubernetes support, designed for enterprise-grade reliability.
+
+**Contribute and Extend:** As an open-source project, community contributions are highly encouraged! If you develop valuable customizations or enhancements, consider contributing them back to the main repository to benefit other DocsGPT users.
+
+<Cards
+      num={3}
+      children={Object.keys(allGuides).map((key, i) => (
+        <Cards.Card
+          key={i}
+          title={allGuides[key].title}
+          href={allGuides[key].href}
+        />
+      ))}
+    />

docs/content/quickstart.mdx

@@ -0,0 +1,122 @@
+---
+title: Quickstart - Launching DocsGPT Web App
+description: Get started with DocsGPT quickly by launching the web application using the setup script.
+---
+
+# Quickstart
+
+**Prerequisites:**
+
+* **Docker:** Ensure you have Docker installed and running on your system.
+
+## Launching DocsGPT (macOS and Linux)
+
+The easiest way to launch DocsGPT is using the provided `setup.sh` script. This script automates the configuration process and offers several setup options.
+
+**Steps:**
+
+1. **Download the DocsGPT Repository:**
+
+   First, you need to download the DocsGPT repository to your local machine. You can do this using Git:
+
+   ```bash
+   git clone https://github.com/arc53/DocsGPT.git
+   cd DocsGPT
+   ```
+
+2. **Run the `setup.sh` script:**
+
+   Navigate to the DocsGPT directory in your terminal and execute the `setup.sh` script:
+
+   ```bash
+   ./setup.sh
+   ```
+
+3. **Follow the interactive setup:**
+
+   The `setup.sh` script will guide you through an interactive menu with the following options:
+
+   ```
+   Welcome to DocsGPT Setup!
+   How would you like to proceed?
+   1) Use DocsGPT Public API Endpoint (simple and free)
+   2) Serve Local (with Ollama)
+   3) Connect Local Inference Engine
+   4) Connect Cloud API Provider
+   5) Advanced: Build images locally (for developers)
+   Choose option (1-5):
+   ```
+
+   Let's break down each option:
+
+   * **1) Use DocsGPT Public API Endpoint (simple and free):**  This is the simplest option to get started. It utilizes the DocsGPT public API, requiring no API keys or local model downloads. Choose this for a quick and easy setup.
+
+   * **2) Serve Local (with Ollama):** This option allows you to run a Large Language Model locally using [Ollama](https://ollama.com/).  You'll be prompted to choose between CPU or GPU for Ollama and select a model to download. This is a good option for local processing and experimentation.
+
+   * **3) Connect Local Inference Engine:**  If you are already running a local inference engine like Llama.cpp, Text Generation Inference (TGI), vLLM, or others, choose this option.  You'll be asked to select your engine and provide the necessary connection details. This is for users with existing local LLM infrastructure.
+
+   * **4) Connect Cloud API Provider:** This option lets you connect DocsGPT to a commercial Cloud API provider such as OpenAI, Google (Vertex AI/Gemini), Anthropic (Claude), Groq, HuggingFace Inference API, or Azure OpenAI. You will need an API key from your chosen provider. Select this if you prefer to use a powerful cloud-based LLM.
+
+   * **5) Modify DocsGPT's source code and rebuild the Docker images locally.** Instead of pulling prebuilt images from Docker Hub or using the hosted/public API, you build the entire backend and frontend from source, customizing how DocsGPT works internally, or run it in an environment without internet access.
+
+   After selecting an option and providing any required information (like API keys or model names), the script will configure your `.env` file and start DocsGPT using Docker Compose.
+
+4. **Access DocsGPT in your browser:**
+
+   Once the setup is complete and Docker containers are running, navigate to [http://localhost:5173/](http://localhost:5173/) in your web browser to access the DocsGPT web application.
+
+5. **Stopping DocsGPT:**
+
+   To stop DocsGPT, simply open a new terminal in the `DocsGPT` directory and run:
+
+   ```bash
+   docker compose -f deployment/docker-compose-hub.yaml down
+   ```
+   (or the specific `docker compose` command shown at the end of the `setup.sh` execution, which may include optional compose files depending on your choices).
+
+## Launching DocsGPT (Windows)
+
+For Windows users, we provide a PowerShell script that offers the same functionality as the macOS/Linux setup script.
+
+**Steps:**
+
+1. **Download the DocsGPT Repository:**
+
+   First, you need to download the DocsGPT repository to your local machine. You can do this using Git:
+
+   ```powershell
+   git clone https://github.com/arc53/DocsGPT.git
+   cd DocsGPT
+   ```
+
+2. **Run the `setup.ps1` script:**
+
+   Execute the PowerShell setup script:
+
+   ```powershell
+   PowerShell -ExecutionPolicy Bypass -File .\setup.ps1
+   ```
+
+3. **Follow the interactive setup:**
+
+   Just like the Linux/macOS script, the PowerShell script will guide you through setting DocsGPT.
+   The script will handle environment configuration and start DocsGPT based on your selections.
+
+4. **Access DocsGPT in your browser:**
+
+   Once the setup is complete and Docker containers are running, navigate to [http://localhost:5173/](http://localhost:5173/) in your web browser to access the DocsGPT web application.
+
+5. **Stopping DocsGPT:**
+
+   To stop DocsGPT run the Docker Compose down command displayed at the end of the setup script's execution.
+
+**Important for Windows:** Ensure Docker Desktop is installed and running correctly on your Windows system before proceeding. The script will attempt to start Docker if it's not running, but you may need to start it manually if there are issues.
+
+**Alternative Method:**
+If you prefer a more manual approach, you can follow our [Docker Deployment documentation](/Deploying/Docker-Deploying) for detailed instructions on setting up DocsGPT on Windows using Docker commands directly.
+
+## Advanced Configuration
+
+For more advanced customization of DocsGPT settings, such as configuring vector stores, embedding models, and other parameters, please refer to the [DocsGPT Settings documentation](/Deploying/DocsGPT-Settings). This guide explains how to modify the `.env` file or `settings.py` for deeper configuration.
+
+Enjoy using DocsGPT!

docs/content/upgrading.mdx

@@ -0,0 +1,66 @@
+---
+title: Upgrading DocsGPT
+description: Upgrade your DocsGPT deployment across Docker Compose, source builds, and Kubernetes.
+---
+
+import { Callout } from 'nextra/components'
+
+# Upgrading DocsGPT
+
+<Callout type="warning">
+  **Upgrading from 0.16.x?** User data moved from MongoDB to Postgres in 0.17.0. Follow the [Postgres Migration guide](/Deploying/Postgres-Migration) before running `docker compose pull` or `git pull` — existing deployments will not start cleanly without it.
+</Callout>
+
+## Check your version
+
+```bash
+docker compose exec backend python -c "from application.version import get_version; print(get_version())"
+```
+
+Release notes: [changelog](/changelog). Tags: [GitHub releases](https://github.com/arc53/DocsGPT/releases).
+
+## Docker Compose — hub images
+
+```bash
+cd DocsGPT/deployment
+docker compose -f docker-compose-hub.yaml pull
+docker compose -f docker-compose-hub.yaml up -d
+```
+
+`pull` fetches the latest image for whichever tag your compose file references. To move to a specific release, edit `image: arc53/docsgpt:<tag>` first.
+
+## Docker Compose — from source
+
+```bash
+cd DocsGPT
+git pull
+docker compose -f deployment/docker-compose.yaml build
+docker compose -f deployment/docker-compose.yaml up -d
+```
+
+Swap `git pull` for `git checkout <tag>` if you want to pin a specific release.
+
+## Kubernetes
+
+```bash
+kubectl set image deployment/docsgpt-backend backend=arc53/docsgpt:<tag>
+kubectl set image deployment/docsgpt-worker worker=arc53/docsgpt:<tag>
+kubectl rollout status deployment/docsgpt-backend
+kubectl rollout status deployment/docsgpt-worker
+```
+
+Full manifests: [Kubernetes deployment guide](/Deploying/Kubernetes-Deploying).
+
+## Migrations
+
+Alembic migrations run on worker startup. To apply manually:
+
+```bash
+docker compose exec backend alembic -c application/alembic.ini upgrade head
+```
+
+`upgrade head` is idempotent.
+
+## Rollback
+
+Set the image tag to the previous release and `up -d` again. Schema changes are not reversible without a backup — take one before upgrading any release that mentions migrations in the changelog.

docs/mdx-components.jsx

@@ -0,0 +1,8 @@
+import { useMDXComponents as getThemeComponents } from 'nextra-theme-docs';
+
+export function useMDXComponents(components) {
+  return {
+    ...getThemeComponents(),
+    ...components,
+  };
+}

docs/next.config.js

@@ -0,0 +1,9 @@
+const nextra = require('nextra').default;
+
+const withNextra = nextra({
+  defaultShowCopyCode: true,
+});
+
+module.exports = withNextra({
+  reactStrictMode: true,
+});

docs/package.json

@@ -0,0 +1,22 @@
+{
+  "scripts": {
+    "dev": "next dev",
+    "build": "next build",
+    "postbuild": "pagefind --site .next/server/app --output-path public/_pagefind",
+    "start": "next start"
+  },
+  "license": "MIT",
+  "dependencies": {
+    "@vercel/analytics": "^1.1.1",
+    "docsgpt-react": "^0.5.1",
+    "next": "^15.5.15",
+    "nextra": "^4.6.1",
+    "nextra-theme-docs": "^4.6.1",
+    "react": "^18.2.0",
+    "react-dom": "^18.2.0"
+  },
+  "devDependencies": {
+    "pagefind": "^1.3.0",
+    "typescript": "^5.9.3"
+  }
+}

docs/public/favicons/site.webmanifest

@@ -0,0 +1,19 @@
+{
+    "name": "",
+    "short_name": "",
+    "icons": [
+        {
+            "src": "/android-chrome-192x192.png",
+            "sizes": "192x192",
+            "type": "image/png"
+        },
+        {
+            "src": "/android-chrome-512x512.png",
+            "sizes": "512x512",
+            "type": "image/png"
+        }
+    ],
+    "theme_color": "#ffffff",
+    "background_color": "#ffffff",
+    "display": "standalone"
+}

docs/public/llms.txt

@@ -0,0 +1,63 @@
+# DocsGPT
+
+> DocsGPT is an open-source platform for building AI agents and assistants with document retrieval, tools, and multi-model support.
+
+This file is a curated map of DocsGPT documentation for LLM and agent use.
+Prioritize Core, Deploying, and Agents for implementation tasks.
+
+## Core
+
+- [Docs Home](https://docs.docsgpt.cloud/): Main documentation landing page.
+- [Quickstart](https://docs.docsgpt.cloud/quickstart): Fastest path to run DocsGPT locally.
+- [Architecture](https://docs.docsgpt.cloud/Guides/Architecture): High-level system architecture.
+- [Development Environment](https://docs.docsgpt.cloud/Deploying/Development-Environment): Backend and frontend local setup.
+- [DocsGPT Settings](https://docs.docsgpt.cloud/Deploying/DocsGPT-Settings): Environment variables and core app configuration.
+
+## Deploying
+
+- [Docker Deployment](https://docs.docsgpt.cloud/Deploying/Docker-Deploying): Run DocsGPT with Docker and Docker Compose.
+- [Kubernetes Deployment](https://docs.docsgpt.cloud/Deploying/Kubernetes-Deploying): Deploy DocsGPT on Kubernetes clusters.
+- [Hosting DocsGPT](https://docs.docsgpt.cloud/Deploying/Hosting-the-app): Hosting overview with cloud options.
+
+## Agents
+
+- [Agent Basics](https://docs.docsgpt.cloud/Agents/basics): Core concepts for building and managing agents.
+- [Workflow Nodes](https://docs.docsgpt.cloud/Agents/nodes): Node types and behavior in agent workflows.
+- [Agent API](https://docs.docsgpt.cloud/Agents/api): Programmatic agent interaction (streaming and non-streaming).
+- [Agent Webhooks](https://docs.docsgpt.cloud/Agents/webhooks): Trigger and automate agents with webhooks.
+
+## Tools
+
+- [Tools Basics](https://docs.docsgpt.cloud/Tools/basics): How tools extend agent capabilities.
+- [Generic API Tool](https://docs.docsgpt.cloud/Tools/api-tool): Configure API calls without custom code.
+- [Creating a Custom Tool](https://docs.docsgpt.cloud/Tools/creating-a-tool): Build custom Python tools for DocsGPT.
+
+## Models
+
+- [Cloud LLM Providers](https://docs.docsgpt.cloud/Models/cloud-providers): Configure hosted model providers.
+- [Local Inference](https://docs.docsgpt.cloud/Models/local-inference): Connect DocsGPT to local inference backends.
+- [Embeddings](https://docs.docsgpt.cloud/Models/embeddings): Select and configure embedding models.
+
+## Extensions
+
+- [API Keys for Integrations](https://docs.docsgpt.cloud/Extensions/api-key-guide): Generate and use DocsGPT API keys.
+- [Chat Widget](https://docs.docsgpt.cloud/Extensions/chat-widget): Embed the DocsGPT chat widget.
+- [Search Widget](https://docs.docsgpt.cloud/Extensions/search-widget): Embed the DocsGPT search widget.
+- [Chrome Extension](https://docs.docsgpt.cloud/Extensions/Chrome-extension): Install and use the browser extension.
+- [Chatwoot Extension](https://docs.docsgpt.cloud/Extensions/Chatwoot-extension): Integrate DocsGPT with Chatwoot.
+
+## Integrations
+
+- [Google Drive Connector](https://docs.docsgpt.cloud/Guides/Integrations/google-drive-connector): Ingest and sync files from Google Drive.
+
+## Optional
+
+- [Customizing Prompts](https://docs.docsgpt.cloud/Guides/Customising-prompts): Template-based prompt customization.
+- [How to Train on Other Documentation](https://docs.docsgpt.cloud/Guides/How-to-train-on-other-documentation): Add additional documentation sources.
+- [Context Compression](https://docs.docsgpt.cloud/Guides/compression): Reduce context while preserving key information.
+- [OCR for Sources and Attachments](https://docs.docsgpt.cloud/Guides/ocr): OCR behavior for ingestion and chat uploads.
+- [How to Use Different LLMs](https://docs.docsgpt.cloud/Guides/How-to-use-different-LLM): Additional model-selection guidance.
+- [Avoiding Hallucinations](https://docs.docsgpt.cloud/Guides/My-AI-answers-questions-using-external-knowledge): Improve answer grounding with external knowledge.
+- [Amazon Lightsail Deployment](https://docs.docsgpt.cloud/Deploying/Amazon-Lightsail): Deploy DocsGPT on AWS Lightsail.
+- [Railway Deployment](https://docs.docsgpt.cloud/Deploying/Railway): Deploy DocsGPT on Railway.
+- [Changelog](https://docs.docsgpt.cloud/changelog): Project release history.

docs/public/toolIcons/tool_api_tool.svg

@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg viewBox="1 6 38 28" xmlns="http://www.w3.org/2000/svg">
+  <path d="M3,33.5c-0.827,0-1.5-0.673-1.5-1.5V8c0-0.827,0.673-1.5,1.5-1.5h34c0.827,0,1.5,0.673,1.5,1.5v24 c0,0.827-0.673,1.5-1.5,1.5H3z" style="fill: rgb(7, 106, 255);"/>
+  <path d="M37,7c0.551,0,1,0.449,1,1v24c0,0.551-0.449,1-1,1H3c-0.551,0-1-0.449-1-1V8c0-0.551,0.449-1,1-1 H37 M37,6H3C1.895,6,1,6.895,1,8v24c0,1.105,0.895,2,2,2h34c1.105,0,2-0.895,2-2V8C39,6.895,38.105,6,37,6L37,6z" style="fill: rgb(7, 106, 255);"/>
+  <path d="M 19.296 13.226 C 20.066 13.06 21.108 12.955 22.147 12.955 C 23.772 12.955 25.153 13.185 26.047 14.038 C 26.88 14.766 27.255 15.931 27.255 17.118 C 27.255 18.638 26.798 19.718 26.07 20.489 C 25.196 21.426 23.801 21.842 22.656 21.842 C 22.47 21.842 22.302 21.842 22.115 21.821 L 22.115 27.045 L 19.297 27.045 L 19.297 13.226 L 19.296 13.226 Z M 22.114 19.616 C 22.259 19.637 22.405 19.637 22.571 19.637 C 23.945 19.637 24.55 18.657 24.55 17.347 C 24.55 16.119 24.049 15.162 22.78 15.162 C 22.532 15.162 22.281 15.203 22.114 15.266 L 22.114 19.616 Z M 29.158 12.955 L 31.976 12.955 L 31.976 27.045 L 29.158 27.045 L 29.158 12.955 Z M 15.001 27.045 L 17.887 27.045 L 14.91 12.955 L 11.342 12.955 L 8.024 27.045 L 10.91 27.045 L 11.524 24.227 L 14.408 24.227 L 15.001 27.045 Z M 13 15.547 L 13.068 15.547 C 13.205 16.467 13.409 17.888 13.568 18.745 L 14.021 21.409 L 11.942 21.409 L 12.457 18.746 C 12.614 17.93 12.841 16.488 13 15.547 Z" style="fill: rgb(255, 255, 255);"/>
+</svg>

docs/public/toolIcons/tool_brave.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 194.18 227.53"><defs><style>.cls-1{fill-rule:evenodd;fill:url(#linear-gradient);}.cls-2{fill:#fff;}</style><linearGradient id="linear-gradient" y1="116.23" x2="194.18" y2="116.23" gradientTransform="matrix(1, 0, 0, -1, 0, 230)" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="#ff5601"/><stop offset="0.5" stop-color="#ff4000"/><stop offset="1" stop-color="#ff1f01"/></linearGradient></defs><g id="Layer_2" data-name="Layer 2"><g id="Layer_1-2" data-name="Layer 1"><path class="cls-1" d="M187.39,54.58l5.34-13.1s-6.8-7.27-15-15.52S152,22.56,152,22.56L132,0H62.14L42.23,22.56S24.76,17.71,16.51,26s-15,15.52-15,15.52L6.8,54.58,0,74s20,75.65,22.33,84.89c4.61,18.19,7.77,25.22,20.88,34.44S80.1,218.55,84,221s8.74,6.56,13.11,6.56,9.22-4.13,13.11-6.56,27.67-18.43,40.78-27.65,16.26-16.25,20.87-34.44C174.19,149.64,194.18,74,194.18,74Z"/><path class="cls-2" d="M121.85,41c2.91,0,24.51-4.12,24.51-4.12S172,67.8,172,74.41c0,5.47-2.21,7.6-4.8,10.12-.54.53-1.1,1.08-1.66,1.67l-19.2,20.37-.63.64c-1.91,1.92-4.73,4.76-2.74,9.47l.41,1c2.18,5.1,4.87,11.39,1.44,17.78-3.64,6.78-9.89,11.31-13.9,10.56s-13.41-5.66-16.87-7.9S99.6,126.8,99.6,123.35c0-2.89,7.88-7.68,11.71-10,.77-.47,1.37-.83,1.71-1.07l1.88-1.18c3.49-2.17,9.8-6.09,10-7.83.2-2.14.12-2.77-2.69-8.06-.6-1.13-1.3-2.33-2-3.58-2.68-4.61-5.69-9.78-5-13.48.75-4.18,7.3-6.57,12.85-8.6l2-.75,5.78-2.17c5.54-2.07,11.69-4.37,12.71-4.84,1.4-.65,1-1.27-3.22-1.67l-2.06-.21c-5.27-.56-15-1.59-19.71-.28l-3.06.84c-5.31,1.43-11.81,3.19-12.44,4.21-.11.18-.22.33-.32.47-.6.85-1,1.41-.32,5,.19,1.08.6,3.19,1.1,5.81,1.46,7.65,3.75,19.58,4,22.26,0,.38.08.74.13,1.09.36,3,.61,5-2.87,5.77l-.91.21c-3.92.9-9.67,2.22-11.75,2.22s-7.83-1.32-11.76-2.22l-.9-.21c-3.48-.79-3.23-2.78-2.87-5.77,0-.35.09-.71.13-1.09.29-2.68,2.58-14.65,4-22.3.5-2.59.9-4.7,1.1-5.77.66-3.6.27-4.16-.33-5-.1-.14-.21-.29-.32-.47-.62-1-7.13-2.78-12.43-4.21l-3.07-.84C66,58.31,56.25,59.34,51,59.9l-2.06.21c-4.26.4-4.62,1-3.22,1.67,1,.47,7.17,2.77,12.71,4.84l5.78,2.17,2,.75c5.55,2,12.1,4.42,12.85,8.6.67,3.7-2.34,8.87-5,13.48-.72,1.25-1.43,2.45-2,3.58-2.82,5.29-2.9,5.92-2.7,8.06.16,1.74,6.47,5.66,10,7.83.82.5,1.48.92,1.88,1.18s.94.6,1.71,1.06c3.83,2.33,11.71,7.13,11.71,10,0,3.45-11,12.49-14.42,14.73S67.3,145.24,63.29,146,53,142.2,49.39,135.42c-3.43-6.38-.74-12.68,1.44-17.78l.41-1c2-4.71-.83-7.55-2.74-9.47l-.63-.64L28.67,86.2c-.56-.59-1.12-1.14-1.66-1.67-2.59-2.52-4.79-4.65-4.79-10.12,0-6.61,25.6-37.53,25.6-37.53S69.42,41,72.33,41c2.33,0,6.82-1.55,11.49-3.16l3.56-1.21a34.33,34.33,0,0,1,9.71-2,34.33,34.33,0,0,1,9.71,2c1.18.39,2.37.81,3.56,1.21C115,39.45,119.52,41,121.85,41Z"/><path class="cls-2" d="M118.14,150.39c4.57,2.35,7.81,4,9,4.78,1.59,1,.62,2.86-.82,3.88s-20.85,16-22.73,17.69l-.76.68c-1.82,1.64-4.13,3.72-5.77,3.72s-4-2.08-5.77-3.72l-.76-.68c-1.88-1.66-21.28-16.67-22.73-17.69s-2.41-2.89-.82-3.88c1.23-.77,4.47-2.44,9-4.79l4.34-2.24c6.84-3.54,15.37-6.54,16.7-6.54s9.86,3,16.7,6.54Z"/></g></g></svg>

docs/public/toolIcons/tool_cryptoprice.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 122.88 122.88"><path d="M17.89 0h88.9c8.85 0 16.1 7.24 16.1 16.1v90.68c0 8.85-7.24 16.1-16.1 16.1H16.1c-8.85 0-16.1-7.24-16.1-16.1v-88.9C0 8.05 8.05 0 17.89 0zm57.04 66.96l16.46 4.96c-1.1 4.61-2.84 8.47-5.23 11.56-2.38 3.1-5.32 5.43-8.85 7-3.52 1.57-8.01 2.36-13.45 2.36-6.62 0-12.01-.96-16.21-2.87-4.19-1.92-7.79-5.3-10.83-10.13-3.04-4.82-4.57-11.02-4.57-18.54 0-10.04 2.67-17.76 8.02-23.17 5.36-5.39 12.93-8.09 22.71-8.09 7.65 0 13.68 1.54 18.06 4.64 4.37 3.1 7.64 7.85 9.76 14.27l-16.55 3.66c-.58-1.84-1.19-3.18-1.82-4.03-1.06-1.43-2.35-2.53-3.86-3.3-1.53-.78-3.22-1.16-5.11-1.16-4.27 0-7.54 1.71-9.8 5.12-1.71 2.53-2.57 6.52-2.57 11.94 0 6.73 1.02 11.33 3.07 13.83 2.05 2.49 4.92 3.73 8.63 3.73 3.59 0 6.31-1 8.15-3.03 1.83-1.99 3.16-4.92 3.99-8.75z" fill-rule="evenodd" clip-rule="evenodd"/></svg>

docs/public/toolIcons/tool_duckduckgo.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 122.88 122.88"><defs><style>.a{fill:#d53;}.b{fill:#fff;}.c{fill:#ddd;}.d{fill:#fc0;}.e{fill:#6b5;}.f{fill:#4a4;}.g{fill:#148;}</style></defs><title>duckduckgo</title><path class="a" d="M122.88,61.44a61.44,61.44,0,1,0-61.44,61.44,61.44,61.44,0,0,0,61.44-61.44Z"/><path class="b" d="M114.37,61.44a52.92,52.92,0,1,0-15.5,37.43,52.76,52.76,0,0,0,15.5-37.43Zm-13.12-39.8A56.29,56.29,0,1,1,61.44,5.15a56.12,56.12,0,0,1,39.81,16.49Z"/><path class="c" d="M43.24,30.15C26.17,34.13,32.43,58,32.43,58l10.81,52.9,4,1.71-4-82.49Zm-4-10.24H34.7L41,22.19s-6.26,0-6.26,4C48.36,25.6,54.61,29,54.61,29l-15.36-9.1Zm0,0Z"/><path class="b" d="M75.66,115.48S62,93.87,62,79.64c0-26.73,17.63-4,17.63-25S62,28.44,62,28.44c-8.53-10.8-25-8.53-25-8.53l4,2.28s-4,1.13-5.12,2.27,10.81-1.7,15.93,2.85C30.72,29,34.13,46.08,34.13,46.08l11.95,68.27,29.58,1.13Zm0,0Z"/><path class="d" d="M75.66,60.87l21.62-5.69C116.62,58,80.78,68.84,78.51,68.27c-17.07-2.85-12,11.37,8.53,6.82s5.12,11.38-13.65,5.12c-26.74-7.39-12.52-20.48,2.27-19.34Z"/><path class="e" d="M70,105.81l1.14-1.7c12.52,4.55,13.09,6.25,12.52-5.12s0-11.38-13.09-1.71c0-2.84-7.39-1.71-8.53,0-11.95-5.12-13.09-6.83-12.52,1.14,1.14,16.5.57,13.65,11.95,8l8.53-.57Zm0,0Z"/><path class="f" d="M60.87,99.56v6.82c.57,1.14,9.67,1.14,9.67-1.14s-4.55,1.71-7.39.57S62,98.42,62,98.42l-1.14,1.14Zm0,0Z"/><path class="g" d="M48.36,43.24c-2.85-3.42-10.24-.57-8.54,4,.57-2.28,4.55-5.69,8.54-4Zm18.2,0c.57-3.42,6.26-4,8-.57a8,8,0,0,0-8,.57Zm-18.77,9.1a1.14,1.14,0,1,1,0,.57v-.57Zm-4.55,2.27a4,4,0,1,0,0-.57v.57Zm29.58-4a1.14,1.14,0,1,1,0,.57v-.57ZM69.4,52.91a3.42,3.42,0,1,0,0-.57v.57Zm0,0Z"/></svg>

docs/public/toolIcons/tool_mcp_tool.svg

@@ -0,0 +1,4 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="64" height="64" fill="none">
+    <path d="M3.49994 11.7501L11.6717 3.57855C12.7762 2.47398 14.5672 2.47398 15.6717 3.57855C16.7762 4.68312 16.7762 6.47398 15.6717 7.57855M15.6717 7.57855L9.49994 13.7501M15.6717 7.57855C16.7762 6.47398 18.5672 6.47398 19.6717 7.57855C20.7762 8.68312 20.7762 10.474 19.6717 11.5785L12.7072 18.543C12.3167 18.9335 12.3167 19.5667 12.7072 19.9572L13.9999 21.2499" stroke="#e3e3e3" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path>
+    <path d="M17.4999 9.74921L11.3282 15.921C10.2237 17.0255 8.43272 17.0255 7.32823 15.921C6.22373 14.8164 6.22373 13.0255 7.32823 11.921L13.4999 5.74939" stroke="#e3e3e3" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"></path>
+</svg>

docs/public/toolIcons/tool_memory.svg

@@ -0,0 +1,3 @@
+<svg xmlns="http://www.w3.org/2000/svg" height="24px" viewBox="0 -960 960 960" width="24px" fill="#e3e3e3">
+  <path d="M240-80q-33 0-56.5-23.5T160-160v-480q0-33 23.5-56.5T240-720h80v-80q0-17 11.5-28.5T360-840q17 0 28.5 11.5T400-800v80h40v-80q0-17 11.5-28.5T480-840q17 0 28.5 11.5T520-800v80h40v-80q0-17 11.5-28.5T600-840q17 0 28.5 11.5T640-800v80h80q33 0 56.5 23.5T800-640v480q0 33-23.5 56.5T720-80H240Zm0-80h480v-480H240v480Zm120-320v-80h240v80H360Zm0 120v-80h240v80H360Zm0 120v-80h160v80H360ZM240-160v-480 480Z"/>
+</svg>

docs/public/toolIcons/tool_notes.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" height="24px" viewBox="0 -960 960 960" width="24px" fill="#e3e3e3"><path d="M320-240h320v-80H320v80Zm0-160h320v-80H320v80ZM240-80q-33 0-56.5-23.5T160-160v-640q0-33 23.5-56.5T240-880h320l240 240v480q0 33-23.5 56.5T720-80H240Zm280-520v-200H240v640h480v-440H520ZM240-800v200-200 640-640Z"/></svg>

docs/public/toolIcons/tool_postgres.svg

@@ -0,0 +1,29 @@
+<?xml version="1.0" encoding="utf-8"?><!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
+<svg width="800px" height="800px" viewBox="-8.78 0 70 70" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:cc="http://creativecommons.org/ns#" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns="http://www.w3.org/2000/svg">
+    <metadata>
+        <rdf:RDF>
+            <cc:Work>
+                <dc:subject>
+                    Data
+                </dc:subject>
+                <dc:identifier>
+                    sql-database-generic
+                </dc:identifier>
+                <dc:title>
+                    SQL Database (Generic)
+                </dc:title>
+                <dc:format>
+                    image/svg+xml
+                </dc:format>
+                <dc:publisher>
+                    Amido Limited
+                </dc:publisher>
+                <dc:creator>
+                    Richard Slater
+                </dc:creator>
+                <dc:type rdf:resource="http://purl.org/dc/dcmitype/StillImage"/>
+            </cc:Work>
+        </rdf:RDF>
+    </metadata>
+    <path d="m 852.97077,1013.9363 c -6.55238,-0.4723 -13.02857,-2.1216 -17.00034,-4.3296 -2.26232,-1.2576 -3.98589,-2.8032 -4.66223,-4.1807 l -0.4024,-0.8196 0,-25.70807 0,-25.7081 0.31843,-0.6465 c 1.42297,-2.889 5.96432,-5.4935 12.30378,-7.0562 2.15195,-0.5305 5.2586,-1.0588 7.79304,-1.3252 2.58797,-0.2721 9.44765,-0.2307 12.02919,0.073 6.86123,0.8061 12.69967,2.6108 16.29768,5.0377 1.38756,0.9359 2.81137,2.4334 3.29371,3.4642 l 0.41358,0.8838 -0.0354,25.6303 -0.0354,25.63047 -0.33195,0.6744 c -0.18257,0.3709 -0.73406,1.1007 -1.22553,1.6216 -2.99181,3.1715 -9.40919,5.5176 -17.8267,6.5172 -1.71567,0.2038 -9.16916,0.3686 -10.92937,0.2417 z m 12.07501,-22.02839 c -0.0252,-0.0657 -1.00472,-0.93831 -2.17671,-1.93922 -1.17199,-1.00091 -2.18138,-1.86687 -2.24309,-1.92436 -0.0617,-0.0575 0.15481,-0.26106 0.48117,-0.45237 0.32635,-0.19131 0.95163,-0.7235 1.3895,-1.18265 1.2805,-1.34272 1.88466,-3.00131 1.88466,-5.17388 0,-2.1388 -0.65162,-3.8645 -1.95671,-5.1818 -1.31533,-1.3278 -2.82554,-1.8983 -5.02486,-1.8983 -3.39007,0 -5.99368,1.9781 -6.82468,5.1851 -0.28586,1.1031 -0.28432,3.33211 0.003,4.31023 0.74941,2.55136 2.79044,4.40434 5.33062,4.83946 0.8596,0.14724 0.97605,0.21071 1.5621,0.85144 0.34829,0.38078 1.06301,1.14085 1.58827,1.68904 l 0.95501,0.9967 2.53878,0 c 1.39633,0 2.51816,-0.0537 2.49296,-0.11939 z m -8.70653,-7.10848 c -0.61119,-0.31868 -0.84225,-0.56599 -1.19079,-1.27453 -0.26919,-0.54724 -0.31522,-0.85851 -0.31824,-2.15197 -0.003,-1.3143 0.0388,-1.5983 0.31987,-2.169 0.45985,-0.9339 1.09355,-1.376 2.07384,-1.4469 1.36454,-0.099 2.15217,0.5707 2.56498,2.1801 0.50612,1.97321 -0.0504,4.07107 -1.26471,4.76729 -0.63707,0.36527 -1.58737,0.40659 -2.18495,0.095 z m -11.25315,3.66269 c 2.66179,-0.5048 4.1728,-2.0528 4.1728,-4.27495 0,-1.97137 -0.97548,-3.12004 -3.6716,-4.32364 -1.54338,-0.689 -2.10241,-1.1215 -2.10241,-1.6268 0,-0.4188 0.53052,-0.8777 1.14813,-0.993 0.60302,-0.1126 2.20237,0.1652 3.14683,0.5467 l 0.79167,0.3198 0,-1.7524 0,-1.7525 -0.85923,-0.1906 c -0.53103,-0.1178 -1.64689,-0.1885 -2.92137,-0.1849 -1.80528,0 -2.15881,0.044 -2.83818,0.3138 -1.98445,0.7878 -2.92613,2.1298 -2.91107,4.1485 0.0141,1.8898 1.01108,3.06864 3.49227,4.12912 1.46399,0.62572 2.05076,1.10218 2.05076,1.66522 0,1.1965 -1.99362,1.34375 -4.10437,0.30315 -0.57805,-0.28498 -1.09739,-0.54137 -1.1541,-0.56976 -0.0567,-0.0284 -0.10311,0.79023 -0.10311,1.81917 0,1.86239 0.002,1.87137 0.33919,1.99974 1.26979,0.48278 4.07626,0.69787 5.52379,0.42335 z m 30.4308,-1.72766 0,-1.58098 -2.40584,0 -2.40583,0 0,-5.43035 0,-5.4303 -2.13089,0 -2.13088,0 0,7.0113 0,7.01131 4.53672,0 4.53672,0 0,-1.58098 z m -14.84745,-27.70503 c 4.23447,-0.2937 7.4086,-0.8482 10.20178,-1.7821 2.78264,-0.9304 4.42643,-2.0562 4.79413,-3.2834 0.14166,-0.4729 0.13146,-0.6523 -0.0665,-1.1708 -0.88775,-2.3245 -5.84694,-4.1104 -13.42493,-4.8345 -3.24154,-0.3098 -9.13671,-0.2094 -12.22745,0.2081 -4.71604,0.6372 -8.54333,1.8208 -10.2451,3.1683 -3.44251,2.726 0.19793,5.7242 8.66397,7.1354 3.67084,0.6119 8.42674,0.828 12.30414,0.559 z" fill="#00bcf2" transform="translate(-830.906 -943.981)"/>
+</svg>

docs/public/toolIcons/tool_read_webpage.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" height="24px" viewBox="0 -960 960 960" width="24px" fill="#e3e3e3"><path d="M480-80q-82 0-155-31.5t-127.5-86Q143-252 111.5-325T80-480q0-83 31.5-155.5t86-127Q252-817 325-848.5T480-880q83 0 155.5 31.5t127 86q54.5 54.5 86 127T880-480q0 82-31.5 155t-86 127.5q-54.5 54.5-127 86T480-80Zm0-82q26-36 45-75t31-83H404q12 44 31 83t45 75Zm-104-16q-18-33-31.5-68.5T322-320H204q29 50 72.5 87t99.5 55Zm208 0q56-18 99.5-55t72.5-87H638q-9 38-22.5 73.5T584-178ZM170-400h136q-3-20-4.5-39.5T300-480q0-21 1.5-40.5T306-560H170q-5 20-7.5 39.5T160-480q0 21 2.5 40.5T170-400Zm216 0h188q3-20 4.5-39.5T580-480q0-21-1.5-40.5T574-560H386q-3 20-4.5 39.5T380-480q0 21 1.5 40.5T386-400Zm268 0h136q5-20 7.5-39.5T800-480q0-21-2.5-40.5T790-560H654q3 20 4.5 39.5T660-480q0 21-1.5 40.5T654-400Zm-16-240h118q-29-50-72.5-87T584-782q18 33 31.5 68.5T638-640Zm-234 0h152q-12-44-31-83t-45-75q-26 36-45 75t-31 83Zm-200 0h118q9-38 22.5-73.5T376-782q-56 18-99.5 55T204-640Z"/></svg>

docs/public/toolIcons/tool_remote_device.svg

@@ -0,0 +1,7 @@
+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="64" height="64" fill="none">
+    <rect x="2.5" y="4" width="19" height="13" rx="2" stroke="#e3e3e3" stroke-width="1.5"/>
+    <path d="M8 21h8" stroke="#e3e3e3" stroke-width="1.5" stroke-linecap="round"/>
+    <path d="M12 17v4" stroke="#e3e3e3" stroke-width="1.5" stroke-linecap="round"/>
+    <path d="M6.5 9l2.5 2-2.5 2" stroke="#e3e3e3" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"/>
+    <path d="M11 13h5" stroke="#e3e3e3" stroke-width="1.5" stroke-linecap="round"/>
+</svg>

docs/public/toolIcons/tool_telegram.svg

@@ -0,0 +1,10 @@
+<svg width="24" height="25" viewBox="0 0 24 25" fill="none" xmlns="http://www.w3.org/2000/svg">
+<path d="M12 0.5C8.81812 0.5 5.76375 1.76506 3.51562 4.01469C1.2652 6.26522 0.000643966 9.31734 0 12.5C0 15.6813 1.26562 18.7357 3.51562 20.9853C5.76375 23.2349 8.81812 24.5 12 24.5C15.1819 24.5 18.2362 23.2349 20.4844 20.9853C22.7344 18.7357 24 15.6813 24 12.5C24 9.31869 22.7344 6.26431 20.4844 4.01469C18.2362 1.76506 15.1819 0.5 12 0.5Z" fill="url(#paint0_linear_5586_9958)"/>
+<path d="M5.43282 12.373C8.93157 10.849 11.2641 9.8443 12.4303 9.3588C15.7641 7.97261 16.4559 7.73186 16.9078 7.7237C17.0072 7.72211 17.2284 7.74667 17.3728 7.86339C17.4928 7.96183 17.5266 8.09495 17.5434 8.18842C17.5584 8.2818 17.5791 8.49461 17.5622 8.66074C17.3822 10.5582 16.6003 15.1629 16.2028 17.2882C16.0359 18.1874 15.7041 18.4889 15.3834 18.5184C14.6859 18.5825 14.1572 18.0579 13.4822 17.6155C12.4266 16.9231 11.8303 16.4922 10.8047 15.8167C9.6197 15.0359 10.3884 14.6067 11.0634 13.9055C11.2397 13.7219 14.3109 10.9291 14.3691 10.6758C14.3766 10.6441 14.3841 10.526 14.3128 10.4637C14.2434 10.4013 14.1403 10.4227 14.0653 10.4395C13.9584 10.4635 12.2728 11.5788 9.00282 13.7851C8.52469 14.114 8.09157 14.2743 7.70157 14.2659C7.27407 14.2567 6.44907 14.0236 5.83595 13.8245C5.08595 13.5802 4.48782 13.451 4.54032 13.036C4.56657 12.82 4.8647 12.599 5.43282 12.373Z" fill="white"/>
+<defs>
+<linearGradient id="paint0_linear_5586_9958" x1="1200" y1="0.5" x2="1200" y2="2400.5" gradientUnits="userSpaceOnUse">
+<stop stop-color="#2AABEE"/>
+<stop offset="1" stop-color="#229ED9"/>
+</linearGradient>
+</defs>
+</svg>

docs/public/toolIcons/tool_todo_list.svg

@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" height="24px" viewBox="0 -960 960 960" width="24px" fill="#e3e3e3"><path d="M240-80q-33 0-56.5-23.5T160-160v-640q0-33 23.5-56.5T240-880h480q33 0 56.5 23.5T800-800v640q0 33-23.5 56.5T720-80H240Zm0-80h480v-640H240v640Zm88-104 56-56-56-56-56 56 56 56Zm0-160 56-56-56-56-56 56 56 56Zm0-160 56-56-56-56-56 56 56 56Zm120 280h232v-80H448v80Zm0-160h232v-80H448v80Zm0-160h232v-80H448v80ZM240-160v-640 640Z"/></svg>

docs/runbooks/sse-notifications.md

@@ -0,0 +1,395 @@
+# SSE Notifications Runbook
+
+> Operations guide for "user says they didn't get a notification" — and
+> the related "the bell never lights up" / "my upload toast hangs" /
+> "the chat answer doesn't reconnect" symptoms.
+
+The user-facing notifications channel is the SSE pipe at
+`/api/events` plus per-message reconnects at
+`/api/messages/<id>/events`. This document maps a user complaint to
+the diagnostic that surfaces the cause.
+
+---
+
+## TL;DR — first 60 seconds
+
+Run these three commands in parallel before anything else:
+
+```bash
+# 1) Is Redis up and serving the pipe? Should print PONG instantly.
+redis-cli -n 2 PING
+
+# 2) Anyone subscribed to the channel right now? Numbers per channel.
+redis-cli -n 2 PUBSUB NUMSUB user:<user_id>
+
+# 3) Is the user's backlog populated? Returns the count of journaled events.
+redis-cli -n 2 XLEN user:<user_id>:stream
+```
+
+- `PING` failing → Redis is the problem. Skip to "Redis-down".
+- `NUMSUB user:<user_id>` returns 0 → no client connected. Skip to "Client never connects".
+- `XLEN user:<user_id>:stream` returns 0 or low → publisher isn't writing. Skip to "Publisher silent".
+- All three look healthy → the events are flowing on the wire; the issue is downstream of the slice (UI rendering, toast suppression, etc.). Skip to "Events flowing but UI silent".
+
+---
+
+## Architecture cheat-sheet
+
+```
+Worker (publish_user_event)             Frontend tab
+        │                                    ▲
+        ▼                                    │  GET /api/events SSE
+  Redis Streams: XADD                  Flask route
+  user:<id>:stream  ──────────────►   replay_backlog (snapshot)
+        │                                    +
+        ▼                              Topic.subscribe (live tail)
+  Redis pub/sub: PUBLISH                     │
+  user:<id>  ────────────────────────────────┘
+```
+
+**Source of truth:**
+- Persistent journal: Redis Stream `user:<user_id>:stream`, capped at
+  `EVENTS_STREAM_MAXLEN` (default 1000) entries via `MAXLEN ~`. ~24h
+  at typical event rates.
+- Live fan-out: Redis pub/sub channel `user:<user_id>`. No durability;
+  subscribers must be attached at publish time.
+
+The chat-stream pipe is separate, parallel infrastructure:
+- Journal: Postgres `message_events` table.
+- Live fan-out: Redis pub/sub `channel:<message_id>`.
+
+Same patterns, different durability layer. This doc covers both;
+they share most diagnostic commands.
+
+---
+
+## Symptom → diagnostic map
+
+### A. "I uploaded a source and the toast never appeared"
+
+User flow: chat → upload → expect toast.
+
+| Step                                              | Command                                                       | Expect                                          |
+| ------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------- |
+| Worker received the task                          | `tail -f celery.log` filtered by user                         | `ingest_worker` start log line                  |
+| Worker published the queued event                 | `redis-cli -n 2 XREVRANGE user:<id>:stream + - COUNT 5`       | A `source.ingest.queued` entry within seconds   |
+| Frontend got it                                   | DevTools → Network → `/api/events` → EventStream tab          | `data: {"type":"source.ingest.queued",...}`     |
+| Slice updated                                     | Redux DevTools → state.upload.tasks                           | Task with matching `sourceId`, `status:'training'` |
+
+If the worker's queued log line is there but the XADD didn't land →
+look for a `publish_user_event payload not JSON-serializable` warning
+in the worker log (the publisher swallows `TypeError`).
+
+If the XADD landed but the frontend never received it → check
+`PUBSUB NUMSUB user:<id>` while the user is on the page. If 0, the
+SSE connection isn't subscribed; skip to "Client never connects".
+
+If the frontend received it but the toast didn't render → the
+`uploadSlice` extraReducer requires `task.sourceId` to match the
+event's `scope.id`. Check the upload route returned `source_id` in
+its POST response (the upload, connector, and reingest paths all
+include it). Idempotent / cached responses must also include
+`source_id` (`_claim_task_or_get_cached`).
+
+### B. "The bell badge never goes up"
+
+There is no bell — the global notifications surface is per-event
+toasts, not an aggregated counter. If the user is on an old build,
+`Cmd-Shift-R` to bypass cache. The surfaces they're looking for are
+`UploadToast` for source uploads and `ToolApprovalToast` for
+tool-approval events.
+
+### C. "My chat answer froze mid-stream and never recovered"
+
+User flow: ask question → answer streaming → network blip → answer
+stops; should reconnect.
+
+```bash
+# Was the original message reserved in PG?
+psql -c "SELECT id, status, prompt FROM conversation_messages \
+  WHERE user_id = '<user>' ORDER BY timestamp DESC LIMIT 5;"
+
+# Did the journal capture events past the user's last-seen seq?
+psql -c "SELECT sequence_no, event_type FROM message_events \
+  WHERE message_id = '<id>' ORDER BY sequence_no;"
+
+# Is the live tail still producing? (subscribe and watch)
+redis-cli -n 2 SUBSCRIBE channel:<message_id>
+```
+
+The frontend should reconnect via `GET /api/messages/<id>/events`
+when the original POST stream closes without a typed `end` or
+`error` event. If it's not reconnecting, `console.warn('Stream
+reconnect failed', ...)` will be in the browser console — the
+reconnect HTTP errored. Common cases:
+
+- The user's JWT rotated mid-stream → 401 on the GET. Frontend
+  doesn't auto-refresh; the user reloads.
+- The user is on a different host than the API and CORS is rejecting
+  the GET → check `application/asgi.py` allow-headers.
+
+### D. "The dev install never delivers any notifications at all"
+
+Default `AUTH_TYPE` unset means `decoded_token = {"sub": "local"}`
+for every request. The SSE client connects without the
+`Authorization` header in this case, and `user:local:stream` is
+the shared channel everything goes to. If the user has multiple dev
+machines pointing at the same Redis, they will see each other's
+events. Confirm with:
+
+```bash
+redis-cli -n 2 KEYS 'user:local:*'
+```
+
+If multiple deployments share the Redis, document that as a known
+multi-user-on-local-channel limitation. Set `AUTH_TYPE=simple_jwt`
+to scope per-user.
+
+### E. "The notifications channel was working, then suddenly stopped after the user reloaded the page"
+
+Likely path: `backlog.truncated` event fired, the slice cleared
+`lastEventId` to null, the closure was carrying the same stale id and
+re-tripped the same truncation on every reconnect. **Verify the user
+is on a current build — `eventStreamClient.ts` must re-read
+`lastEventId = opts.getLastEventId();` without a truthy guard so the
+null clear propagates into the next reconnect.**
+
+### F. "I keep getting 429 on /api/events"
+
+The per-user concurrent-connection cap (`SSE_MAX_CONCURRENT_PER_USER`,
+default 8) refused the connection. User has too many tabs open or a
+runaway reconnect loop. `redis-cli -n 2 GET user:<id>:sse_count`
+shows the live counter; the TTL is 1h from the last connection
+attempt (rolling — every INCR re-seeds it), so the key only ages
+out after the user stops reconnecting for a full hour.
+
+If the count is wedged high without explanation, the
+counter-DECR-in-finally path didn't run (worker SIGKILL, OOM). Wait
+for the TTL or `redis-cli -n 2 DEL user:<id>:sse_count` to reset.
+
+### G. "Replay snapshot stops at 200 events"
+
+The route caps each replay at `EVENTS_REPLAY_MAX_PER_REQUEST`
+(default 200). The cap is intentionally **silent** — the route does NOT
+emit a `backlog.truncated` notice for cap-hit. The 200 entries each
+carry their own `id:` header, so the frontend's slice cursor
+advances to the most-recent delivered id. Next reconnect sends
+`last_event_id=<max_replayed>` and the snapshot resumes from there.
+A user that was 1000 entries behind catches up over ~5 reconnects.
+
+If the user reports getting HTTP 429 on `/api/events` despite being
+well under `SSE_MAX_CONCURRENT_PER_USER`, they hit the windowed
+replay budget (`EVENTS_REPLAY_BUDGET_REQUESTS_PER_WINDOW`, default
+30 / `EVENTS_REPLAY_BUDGET_WINDOW_SECONDS` 60s). The route refuses
+the connection so the slice cursor stays pinned at whatever value
+it had; the frontend backs off and the next reconnect (after the
+window rolls) gets the proper snapshot. Serving the live tail
+without a snapshot used to be the behavior here, but that let the
+client advance `lastEventId` past entries it never received,
+permanently stranding the un-replayed window — so the route now
+429s instead. `redis-cli -n 2 GET user:<id>:replay_count` shows the
+current counter; TTL is the window size.
+
+`backlog.truncated` is emitted ONLY when the client's
+`Last-Event-ID` has slid off the MAXLEN'd window — i.e. the journal
+is genuinely gone past the cursor and the frontend should clear the
+slice cursor and refetch state. Treating cap-hit or
+budget-exhaustion the same way would lock the user into re-receiving
+the oldest 200 entries on every reconnect (the cursor would clear,
+the snapshot would re-serve from the start, the cap would re-trip).
+
+### H. "User says push notifications stopped after a deploy"
+
+- Pull `event.published topic=user:<id> type=...` from the worker
+  logs to confirm the publisher is still firing.
+- Pull `event.connect user=<id>` from the API logs to confirm the
+  client is reconnecting.
+- Check the gunicorn worker count and `WSGIMiddleware(workers=32)` —
+  if the deploy reduced worker count, the per-user cap is still 8
+  but total concurrent SSE connections are bounded by `gunicorn
+  workers × 32`. A capacity miss looks like users randomly getting
+  429'd.
+
+---
+
+## Common failure modes
+
+### Redis-down
+
+Symptoms: `/api/events` returns 200 but emits only `: connected`
+then the body closes. `XLEN` and `PUBLISH` both fail. The publisher's
+`record_event` swallows the failure and returns False; the live tail
+publish also drops on the floor. Frontend retries forever with
+exponential backoff.
+
+Resolution: bring Redis back. The journal is gone (was in-memory
+only — Streams persist within a single Redis instance, no replication
+configured). New events flow as soon as Redis comes back.
+
+### `AUTH_TYPE` misconfigured = sub:"local" cross-stream
+
+Symptoms: every user shares `user:local:stream`. Any user sees
+everyone else's notifications.
+
+Resolution: set `AUTH_TYPE=simple_jwt` (or `session_jwt`) in `.env`.
+The events route logs a one-time WARNING per process when
+`sub == "local"` is observed. A repeat WARNING after a restart
+confirms the misconfiguration.
+
+### MAXLEN trimmed past Last-Event-ID
+
+Symptoms: client reconnects with `last_event_id=X`, snapshot returns
+the entire MAXLEN'd backlog (because X is older than the oldest
+retained entry). Old events appear duplicated.
+
+Detection: the route's `_oldest_retained_id` check emits
+`backlog.truncated` when this case fires. Frontend's
+`dispatchSSEEvent` clears `lastEventId` so the next reconnect starts
+fresh.
+
+If the WARNING isn't firing but symptoms match: the user's client
+may have a corrupt cached `lastEventId`. `localStorage` doesn't
+store this state; check Redux state via DevTools.
+
+### Stale event-stream client
+
+Symptoms: events visible in `XRANGE` but the frontend slice doesn't
+update.
+
+```bash
+# Is the client subscribed?
+redis-cli -n 2 PUBSUB NUMSUB user:<id>
+
+# When did its connection start?
+grep "event.connect user=<id>" /var/log/docsgpt.log | tail -3
+```
+
+If `NUMSUB` is 0 and no recent `event.connect`, the user's tab is
+closed or the connection died and never reconnected. Push them to
+reload.
+
+### Publisher silent
+
+Symptoms: worker is processing the task (Celery says SUCCESS), but
+no XADD and no PUBLISH. User sees no events.
+
+```bash
+# Was the publisher import error suppressed?
+grep "publish_user_event" /var/log/celery.log | grep -i "warn\|error" | tail -20
+
+# Is push disabled?
+grep "ENABLE_SSE_PUSH" /var/log/docsgpt.log | tail -5
+```
+
+`ENABLE_SSE_PUSH=False` in `.env` would silence the publisher
+globally. Useful for incident response if a runaway publisher is
+DoS'ing Redis; toggle off, fix root cause, toggle on.
+
+---
+
+## Useful one-liners
+
+```bash
+# Watch a user's live event stream in real time (all events, all types)
+redis-cli -n 2 PSUBSCRIBE 'user:*' | grep "user:<id>"
+
+# Last 10 events the user would see on reconnect
+redis-cli -n 2 XREVRANGE user:<id>:stream + - COUNT 10
+
+# Live count of subscribed clients per user
+redis-cli -n 2 PUBSUB NUMSUB $(redis-cli -n 2 PUBSUB CHANNELS 'user:*')
+
+# Trim a runaway stream (CAREFUL — destroys backlog for all current
+# subscribers; OK after explaining to the user)
+redis-cli -n 2 XTRIM user:<id>:stream MAXLEN 0
+
+# Clear a wedged concurrent-connection counter
+redis-cli -n 2 DEL user:<id>:sse_count
+
+# Force-flip every client to re-snapshot (drop the stream key entirely
+# — destroys the backlog; clients reconnect with their last id and
+# get a backlog.truncated)
+redis-cli -n 2 DEL user:<id>:stream
+```
+
+---
+
+## Settings reference
+
+Everything in `application/core/settings.py`:
+
+| Setting                                       | Default | Purpose                                       |
+| --------------------------------------------- | ------- | --------------------------------------------- |
+| `ENABLE_SSE_PUSH`                             | `True`  | Master switch. False = publisher no-ops, route serves "push_disabled" comment. |
+| `EVENTS_STREAM_MAXLEN`                        | `1000`  | Per-user backlog cap. Approximate via `XADD MAXLEN ~`. |
+| `SSE_KEEPALIVE_SECONDS`                       | `15`    | Comment-frame cadence. Must sit under reverse-proxy idle close. |
+| `SSE_MAX_CONCURRENT_PER_USER`                 | `8`     | Cap on simultaneous SSE connections per user. 0 = disabled. |
+| `EVENTS_REPLAY_MAX_PER_REQUEST`               | `200`   | Hard cap on snapshot rows per request. |
+| `EVENTS_REPLAY_BUDGET_REQUESTS_PER_WINDOW`    | `30`    | Per-user replays per window. 0 = disabled. |
+| `EVENTS_REPLAY_BUDGET_WINDOW_SECONDS`         | `60`    | Window length. |
+| `MESSAGE_EVENTS_RETENTION_DAYS`               | `14`    | Retention for the `message_events` journal; `cleanup_message_events` beat task deletes older rows. |
+
+---
+
+## Known limitations
+
+### Each tab runs its own SSE connection
+
+There is no cross-tab dedup. Every tab open to the app holds its
+own SSE connection and dispatches every received event into its
+own Redux store, so a user with N tabs open will see N copies of
+each toast. With `SSE_MAX_CONCURRENT_PER_USER=8` (the default) a
+heavy multi-tab user can also hit the connection cap and start
+seeing 429s. Cross-tab dedup via a `BroadcastChannel` ring +
+`navigator.locks`-based leader election is tracked as future work.
+
+### `/c/<unknown-id>` normalises to `/c/new`
+
+If a user navigates to a conversation id that isn't in their
+loaded list, the conversation route rewrites the URL to `/c/new`.
+`ToolApprovalToast`'s gate uses `useMatch('/c/:conversationId')`,
+so for the brief window after the rewrite the toast may surface
+for a conversation the user *thought* they were already viewing.
+Pre-existing route behaviour; not a notifications regression.
+
+### Terminal events un-dismiss running uploads
+
+`frontend/src/upload/uploadSlice.ts` sets `dismissed: false` when
+an upload reaches `completed` or `failed`. If the user dismissed a
+running task and the terminal SSE arrives later, the toast pops
+back. Intentional ("notify the user it's done"); revisit if the
+re-surface UX is too aggressive for v2.
+
+### Reconnect reader needs the ASGI entrypoint
+
+The chat-stream reconnect reader `GET /api/messages/<id>/events`
+is a native-async Starlette route mounted in
+`application/asgi.py`, not a Flask route. Plain `flask run`
+serves only the WSGI Flask app, so under it that endpoint 404s
+and reconnect-after-disconnect can't resume. Run the backend via
+`uvicorn application.asgi:asgi_app --reload` (or the production
+gunicorn uvicorn-worker) to exercise it.
+
+### Werkzeug doesn't auto-reload route files
+
+The dev server (`flask run`) doesn't watch
+`application/api/events/routes.py` for changes by default.
+After editing the route, restart Flask manually — `--reload`
+isn't on. (Production gunicorn reloads via deploy.)
+
+### MCP OAuth completion can fall outside the user stream's MAXLEN window
+
+`get_oauth_status` scans up to `EVENTS_STREAM_MAXLEN` (~1000) entries via `XREVRANGE`. If the user has a high-rate ingest running concurrent with the OAuth handshake, the `mcp.oauth.completed` envelope can be trimmed off the back before they click Save. Symptom: backend returns "OAuth failed or not completed" even though the popup completed successfully.
+
+Mitigation today: bump `EVENTS_STREAM_MAXLEN` per-deployment if your users routinely flood the channel during OAuth flows. A dedicated short-TTL Redis key for OAuth task results is tracked as a follow-up.
+
+### React StrictMode double-mounts SSE
+
+In dev, React 18 StrictMode mounts → unmounts → remounts every
+component, briefly opening two SSE connections per tab before the
+first is aborted. With `SSE_MAX_CONCURRENT_PER_USER=8` and 4–5
+tabs open concurrently you can transiently hit the cap and see
+HTTP 429 on cold-load. The first connection's counter increment
+fires before the AbortController-induced disconnect can decrement
+it. Production (single mount, no StrictMode) is unaffected; raise
+the cap in dev or accept transient 429s.