chore: import upstream snapshot with attribution

docs/scripts/README.md

@@ -0,0 +1,228 @@
+# CUA Documentation Scripts
+
+This directory contains scripts for crawling, indexing, and serving CUA documentation through a Model Context Protocol (MCP) server.
+
+## Scripts
+
+### Local Scripts
+
+- **crawl_docs.py**: Crawls cua.ai/docs using Playwright
+- **generate_db.py**: Creates LanceDB vector database for semantic search
+- **generate_sqlite.py**: Creates SQLite FTS5 database for full-text search
+
+### Modal Deployment
+
+- **modal_app.py**: Complete Modal app with scheduled crawling and MCP server deployment
+
+## Installation
+
+Install dependencies using uv:
+
+```bash
+# From the repository root
+uv sync --group docs-scripts
+
+# crawl_docs.py drives a headless Chromium via Playwright; install the browser
+# binary once (the pip package alone does not include it). The Modal image
+# installs this automatically.
+uv run playwright install chromium
+```
+
+## Usage
+
+### Option 1: Local Development
+
+#### 1. Crawl Documentation
+
+```bash
+uv run docs/scripts/crawl_docs.py
+```
+
+#### 2. Generate Databases
+
+```bash
+# Generate vector database for semantic search
+uv run docs/scripts/generate_db.py
+
+# Generate SQLite FTS5 database for full-text search
+uv run docs/scripts/generate_sqlite.py
+```
+
+### Option 2: Modal Deployment (Production)
+
+The Modal app provides a production-ready deployment with:
+
+- **Scheduled daily crawling** at 6 AM UTC
+- **Persistent storage** using Modal volumes
+- **Scalable MCP server** with automatic database regeneration
+
+#### Initial Setup
+
+1. Install Modal CLI:
+
+```bash
+pip install modal
+```
+
+2. Authenticate with Modal:
+
+```bash
+modal setup
+```
+
+#### Deploy to Modal
+
+```bash
+# Initial deployment with data generation
+modal run docs/scripts/modal_app.py
+
+# Deploy the app (includes scheduled crawling + MCP server)
+modal deploy docs/scripts/modal_app.py
+```
+
+#### Access the MCP Server
+
+After deployment, Modal will provide a public URL for the MCP server:
+
+```
+https://your-username--cua-docs-mcp-web.modal.run/mcp/
+```
+
+Use this URL with the MCP Inspector or any MCP client:
+
+```bash
+npx @modelcontextprotocol/inspector
+# Enter URL: https://your-username--cua-docs-mcp-web.modal.run/mcp/
+# Transport: Streamable HTTP
+```
+
+#### Monitor Scheduled Crawls
+
+View scheduled crawl runs in the Modal dashboard:
+
+```bash
+modal app show cua-docs-mcp
+```
+
+The crawler runs daily at 6 AM UTC and automatically updates the databases.
+
+## Code Indexing
+
+The Modal app also indexes the CUA source code across all git tags, enabling semantic and full-text search over versioned code.
+
+### Architecture
+
+Code indexing uses **parallel sharded processing** for performance:
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                  generate_code_index_parallel()                  │
+│                                                                 │
+│  1. Clone/fetch git repository                                  │
+│  2. Get all tags (e.g., agent-v0.7.3, computer-v0.5.0)         │
+│  3. Group tags by component                                     │
+│  4. Dispatch parallel workers via Modal starmap                 │
+└─────────────────────────────────────────────────────────────────┘
+                              │
+           ┌──────────────────┼──────────────────┐
+           ▼                  ▼                  ▼
+    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
+    │ index_      │    │ index_      │    │ index_      │
+    │ component   │    │ component   │    │ component   │
+    │ (agent)     │    │ (computer)  │    │ (lume)      │
+    │             │    │             │    │             │
+    │ 112 tags    │    │ 58 tags     │    │ 49 tags     │
+    └─────────────┘    └─────────────┘    └─────────────┘
+           │                  │                  │
+           ▼                  ▼                  ▼
+    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
+    │ SQLite +    │    │ SQLite +    │    │ SQLite +    │
+    │ LanceDB     │    │ LanceDB     │    │ LanceDB     │
+    │ (agent)     │    │ (computer)  │    │ (lume)      │
+    └─────────────┘    └─────────────┘    └─────────────┘
+```
+
+Each component gets its own databases:
+
+- `code_index_{component}.sqlite` - FTS5 full-text search
+- `code_index_{component}.lancedb/` - Vector embeddings for semantic search
+
+### Running Code Indexing
+
+```bash
+# Run parallel code indexing (default)
+modal run docs/scripts/modal_app.py --code-only
+
+# Run in detached mode to monitor via dashboard
+modal run --detach docs/scripts/modal_app.py --code-only
+
+# Run sequential (legacy) mode
+modal run docs/scripts/modal_app.py --code-only --no-parallel
+
+# Skip code indexing, only crawl docs
+modal run docs/scripts/modal_app.py --skip-code
+```
+
+### MCP Server: Querying Sharded Databases
+
+The MCP server automatically discovers and queries across all component databases:
+
+**SQLite Queries** - Uses `ATTACH DATABASE` to create a unified view:
+
+```sql
+-- This queries across ALL component databases
+SELECT component, version, file_path
+FROM code_files
+WHERE component = 'agent' AND version = '0.7.3'
+
+-- Full-text search across all components
+SELECT * FROM code_files_fts
+WHERE code_files_fts MATCH 'ComputerAgent'
+```
+
+**Vector Search** - Queries all LanceDBs and merges results by similarity:
+
+```python
+# Searches all component databases, returns top results
+query_code_vectors("screenshot capture implementation", limit=10)
+
+# Search specific component only
+query_code_vectors("agent loop", component="agent", limit=10)
+```
+
+### Database Schema
+
+**SQLite Table: code_files**
+| Column | Type | Description |
+|--------|------|-------------|
+| id | INTEGER | Primary key |
+| component | TEXT | Component name (agent, computer, etc.) |
+| version | TEXT | Version string (0.7.3) |
+| file_path | TEXT | Path within repository |
+| content | TEXT | Full source code |
+| language | TEXT | python, typescript, javascript |
+
+**LanceDB Schema: code**
+| Column | Type | Description |
+|--------|------|-------------|
+| text | TEXT | Source code (embedded) |
+| vector | VECTOR(384) | all-MiniLM-L6-v2 embedding |
+| component | TEXT | Component name |
+| version | TEXT | Version string |
+| file_path | TEXT | Path within repository |
+| language | TEXT | Programming language |
+
+### Size Limits
+
+- **SQLite**: Files up to 1MB are indexed
+- **LanceDB**: Files up to 100KB are embedded (larger files skip embedding)
+- **File types**: `.py`, `.ts`, `.tsx`, `.js`
+
+### Scheduled Indexing
+
+Code indexing runs daily at 5 AM UTC (before docs crawl at 6 AM):
+
+```bash
+# View scheduled runs
+modal app show cua-docs-mcp
+```

docs/scripts/check-hygiene.ts

@@ -0,0 +1,52 @@
+#!/usr/bin/env npx tsx
+
+import * as fs from 'node:fs/promises';
+import * as path from 'node:path';
+import fg from 'fast-glob';
+
+const DOCS_DIR = path.resolve(__dirname, '..');
+const CONTENT_DIR = path.join(DOCS_DIR, 'content/docs');
+
+const bannedPatterns: Array<[RegExp, string]> = [
+  [/\bSTOLE FOCUS\b/, 'internal modality recorder verdict'],
+  [/\bax-bg\b|\bpx-bg\b|\bpx-fg\b|\bax-fg\b/, 'internal modality recorder lane'],
+  [/\bderec\.sh\b/, 'internal test harness script'],
+  [/\baz exec\b/, 'internal CI/container access detail'],
+  [/\bTEST_SUITE\.md\b|\bFINDINGS\.md\b/, 'repo-side contributor document reference'],
+  [/\bNousResearch\b|#47065\b|#22865\b/, 'private partner or issue reference'],
+  [/\bdocumented by a separate agent\b/i, 'authoring note'],
+  [/\bDo not imply\b/i, 'authoring instruction'],
+  [/\bdiorama\b/i, 'misnamed docs framework'],
+];
+
+async function main() {
+  const files = await fg('**/*.mdx', { cwd: CONTENT_DIR });
+  const failures: string[] = [];
+
+  for (const file of files) {
+    const abs = path.join(CONTENT_DIR, file);
+    const content = await fs.readFile(abs, 'utf8');
+    const lines = content.split(/\r?\n/);
+
+    for (const [lineIndex, line] of lines.entries()) {
+      for (const [pattern, reason] of bannedPatterns) {
+        if (pattern.test(line)) {
+          failures.push(`${file}:${lineIndex + 1}: ${reason}: ${line.trim()}`);
+        }
+      }
+    }
+  }
+
+  if (failures.length > 0) {
+    console.error('Public docs hygiene check failed:\n');
+    console.error(failures.join('\n'));
+    process.exit(1);
+  }
+
+  console.log('Public docs hygiene check passed.');
+}
+
+main().catch((error) => {
+  console.error('Fatal error:', error);
+  process.exit(1);
+});

docs/scripts/check-links.ts

@@ -0,0 +1,98 @@
+#!/usr/bin/env npx tsx
+
+/**
+ * Documentation Link Checker
+ *
+ * Uses next-validate-link (by the Fumadocs author) to validate all internal
+ * links across MDX documentation files. Builds the valid URL map from the
+ * content directory and validates links using remark-based MDX parsing.
+ *
+ * Usage:
+ *   pnpm docs:check-links                # Check internal links
+ *   pnpm docs:check-links:external       # Also check external links
+ */
+
+import { readFiles, validateFiles, printErrors, type ScanResult } from 'next-validate-link';
+import * as path from 'path';
+import fg from 'fast-glob';
+
+const DOCS_DIR = path.resolve(__dirname, '..');
+const CONTENT_DIR = path.join(DOCS_DIR, 'content/docs');
+
+/**
+ * Build the ScanResult (valid URL map) directly from content files.
+ * This is more reliable than scanURLs for catch-all routes.
+ */
+function buildScannedUrls(): ScanResult {
+  const urls = new Map<string, {}>();
+  const mdxFiles = fg.sync('**/*.mdx', { cwd: CONTENT_DIR });
+
+  for (const file of mdxFiles) {
+    const slug = file.replace(/\.mdx$/, '').replace(/\/index$/, '');
+    urls.set(`/${slug}`, {});
+  }
+
+  // Also add the root
+  urls.set('/', {});
+
+  return { urls, fallbackUrls: [] };
+}
+
+function pathToUrl(filePath: string): string | undefined {
+  const slug = filePath
+    .replace(/^content\/docs\//, '')
+    .replace(/\.mdx$/, '')
+    .replace(/\/index$/, '');
+  return `/${slug}`;
+}
+
+async function main() {
+  const args = process.argv.slice(2);
+  const checkExternal = args.includes('--external');
+
+  const scanned = buildScannedUrls();
+
+  // Read all MDX content files
+  const files = await readFiles('content/docs/**/*.mdx', { pathToUrl });
+
+  // Validate all links in the MDX files
+  const results = await validateFiles(files, {
+    scanned,
+    checkExternal,
+    // Fumadocs resolves relative MDX links at runtime, skip them here
+    checkRelativePaths: false,
+    checkRelativeUrls: false,
+    pathToUrl,
+    whitelist: (url: string) => {
+      const pathname = url.split(/[#?]/)[0];
+      const ext = path.extname(pathname).toLowerCase();
+      const staticExts = new Set([
+        '.png',
+        '.jpg',
+        '.jpeg',
+        '.gif',
+        '.svg',
+        '.ico',
+        '.webp',
+        '.avif',
+        '.mp4',
+        '.webm',
+        '.pdf',
+        '.css',
+        '.js',
+        '.woff',
+        '.woff2',
+        '.ttf',
+        '.eot',
+      ]);
+      return staticExts.has(ext);
+    },
+  });
+
+  printErrors(results, true);
+}
+
+main().catch((err) => {
+  console.error('Fatal error:', err);
+  process.exit(1);
+});

docs/scripts/crawl_docs.py

@@ -0,0 +1,389 @@
+"""
+Comprehensive crawler for cua.ai/docs using Playwright.
+Recursively crawls all documentation pages and saves content to JSON files.
+"""
+
+import asyncio
+import html
+from html.parser import HTMLParser
+import json
+import re
+from pathlib import Path
+from urllib.parse import urljoin, urlparse
+
+from playwright.async_api import Browser, async_playwright
+
+# Configuration
+BASE_URL = "https://cua.ai"
+DOCS_URL = f"{BASE_URL}/docs"
+OUTPUT_DIR = Path(__file__).parent.parent / "crawled_data"
+MAX_CONCURRENT = 5  # Limit concurrent requests to be polite
+DELAY_BETWEEN_REQUESTS = 0.5  # seconds
+
+
+class HTMLToMarkdown(HTMLParser):
+    """Small dependency-free HTML-to-Markdown converter for crawled docs pages.
+
+    Extraction is scoped to the page's main content container (``<article>``,
+    falling back to ``<main>``) and site chrome (``nav``/``aside``/``footer``) is
+    dropped, so the crawled corpus is the documentation body rather than the
+    navigation tree that repeats identically on every page.
+    """
+
+    block_tags = {
+        "blockquote",
+        "br",
+        "div",
+        "h1",
+        "h2",
+        "h3",
+        "h4",
+        "h5",
+        "h6",
+        "header",
+        "li",
+        "main",
+        "ol",
+        "p",
+        "pre",
+        "section",
+        "table",
+        "tr",
+        "ul",
+    }
+    # Content of these tags is dropped entirely: non-text assets and the site
+    # chrome (sidebar/nav tree, "on this page" aside, footer) that is identical
+    # on every page and would otherwise dominate the embedded corpus.
+    skip_tags = {"script", "style", "svg", "nav", "aside", "footer"}
+
+    def __init__(self, scope_tag: str | None = None) -> None:
+        super().__init__(convert_charrefs=True)
+        self.parts: list[str] = []
+        self.skip_depth = 0
+        self.in_pre = False
+        # When set, only emit text while inside this container; None = emit all.
+        self.scope_tag = scope_tag
+        self.scope_depth = 0
+
+    @property
+    def _capturing(self) -> bool:
+        return self.scope_tag is None or self.scope_depth > 0
+
+    def handle_starttag(self, tag: str, attrs: list[tuple[str, str | None]]) -> None:
+        if tag in self.skip_tags:
+            self.skip_depth += 1
+            return
+        if tag == self.scope_tag:
+            self.scope_depth += 1
+        if self.skip_depth or not self._capturing:
+            return
+        if tag in self.block_tags:
+            self.parts.append("\n")
+        if tag == "li":
+            self.parts.append("- ")
+        elif tag == "pre":
+            self.in_pre = True
+            self.parts.append("\n```\n")
+        elif tag == "code" and not self.in_pre:
+            self.parts.append("`")
+
+    def handle_endtag(self, tag: str) -> None:
+        if tag in self.skip_tags and self.skip_depth:
+            self.skip_depth -= 1
+            return
+        if self.skip_depth:
+            return
+        if self._capturing:
+            if tag == "pre":
+                self.in_pre = False
+                self.parts.append("\n```\n")
+            elif tag == "code" and not self.in_pre:
+                self.parts.append("`")
+            if tag in self.block_tags:
+                self.parts.append("\n")
+        if tag == self.scope_tag and self.scope_depth:
+            self.scope_depth -= 1
+
+    def handle_data(self, data: str) -> None:
+        if self.skip_depth or not self._capturing:
+            return
+        text = data if self.in_pre else re.sub(r"\s+", " ", data)
+        if text.strip():
+            self.parts.append(text)
+
+    def markdown(self) -> str:
+        text = html.unescape("".join(self.parts))
+        text = re.sub(r"[ \t]+\n", "\n", text)
+        text = re.sub(r"\n{3,}", "\n\n", text)
+        return text.strip()
+
+
+def html_to_markdown(page_html: str) -> str:
+    # Prefer the main content container so the navigation/sidebar chrome that
+    # repeats on every page does not pollute the crawled corpus; fall back to
+    # the whole document when neither container is present.
+    scope_tag = None
+    for tag in ("article", "main"):
+        if re.search(rf"<{tag}[\s>]", page_html, re.IGNORECASE):
+            scope_tag = tag
+            break
+    parser = HTMLToMarkdown(scope_tag)
+    parser.feed(page_html)
+    return parser.markdown()
+
+
+def extract_metadata(page_html: str, title: str) -> dict[str, str]:
+    description = ""
+    match = re.search(
+        r'<meta[^>]+name=["\']description["\'][^>]+content=["\']([^"\']*)["\']',
+        page_html,
+        re.IGNORECASE,
+    )
+    if match:
+        description = html.unescape(match.group(1))
+    return {"title": title, "description": description}
+
+
+class CuaDocsCrawler:
+    def __init__(self):
+        self.visited_urls: set[str] = set()
+        self.to_visit: set[str] = set()
+        self.failed_urls: set[str] = set()
+        self.all_data: list[dict] = []
+        self.semaphore = asyncio.Semaphore(MAX_CONCURRENT)
+
+    def normalize_url(self, url: str) -> str:
+        """Normalize URL to avoid duplicates"""
+        parsed = urlparse(url)
+        # Remove trailing slashes and fragments
+        path = parsed.path.rstrip("/")
+        if not path:
+            path = ""
+        return f"{parsed.scheme}://{parsed.netloc}{path}"
+
+    def is_valid_url(self, url: str) -> bool:
+        """Check if URL should be crawled (only /docs pages)"""
+        parsed = urlparse(url)
+
+        # Only crawl cua.ai pages
+        if parsed.netloc and parsed.netloc not in ["cua.ai", "www.cua.ai"]:
+            return False
+
+        # Only crawl /docs paths
+        if not parsed.path.startswith("/docs"):
+            return False
+
+        # Skip non-page resources
+        skip_extensions = [
+            ".pdf",
+            ".png",
+            ".jpg",
+            ".jpeg",
+            ".gif",
+            ".svg",
+            ".css",
+            ".js",
+            ".ico",
+            ".woff",
+            ".woff2",
+            ".ttf",
+            ".zip",
+            ".tar",
+            ".gz",
+        ]
+        if any(parsed.path.lower().endswith(ext) for ext in skip_extensions):
+            return False
+
+        # Skip external links and anchors
+        if url.startswith("#") or url.startswith("mailto:") or url.startswith("javascript:"):
+            return False
+
+        return True
+
+    def extract_links(self, html: str, current_url: str) -> set[str]:
+        """Extract all internal links from HTML content"""
+        links = set()
+
+        # Find all href attributes
+        href_pattern = r'href=["\']([^"\']+)["\']'
+        matches = re.findall(href_pattern, html, re.IGNORECASE)
+
+        for href in matches:
+            # Convert relative URLs to absolute
+            if href.startswith("/"):
+                full_url = urljoin(BASE_URL, href)
+            elif href.startswith("http"):
+                full_url = href
+            elif not href.startswith("#") and not href.startswith("mailto:"):
+                full_url = urljoin(current_url, href)
+            else:
+                continue
+
+            normalized = self.normalize_url(full_url)
+            if self.is_valid_url(normalized):
+                links.add(normalized)
+
+        return links
+
+    def extract_path_info(self, url: str) -> dict:
+        """Extract meaningful path information from URL"""
+        parsed = urlparse(url)
+        path = parsed.path.replace("/docs/", "").strip("/")
+        parts = path.split("/") if path else []
+
+        return {
+            "path": path,
+            "category": parts[0] if parts else "root",
+            "subcategory": parts[1] if len(parts) > 1 else None,
+            "page": parts[-1] if parts else "index",
+            "depth": len(parts),
+        }
+
+    async def crawl_page(self, browser: Browser, url: str) -> dict | None:
+        """Crawl a single page"""
+        async with self.semaphore:
+            page = None
+            try:
+                print(f"Crawling: {url}")
+
+                page = await browser.new_page()
+                response = await page.goto(url, wait_until="networkidle", timeout=30_000)
+                if response is None or not response.ok:
+                    status = response.status if response else "no response"
+                    print(f"Failed to crawl {url}: HTTP {status}")
+                    self.failed_urls.add(url)
+                    return None
+
+                page_html = await page.content()
+                metadata = extract_metadata(page_html, await page.title())
+
+                # Extract new links from the page
+                new_links = self.extract_links(page_html, url)
+                for link in new_links:
+                    if link not in self.visited_urls and link not in self.to_visit:
+                        self.to_visit.add(link)
+
+                path_info = self.extract_path_info(url)
+
+                page_data = {
+                    "url": url,
+                    "title": metadata["title"],
+                    "description": metadata["description"],
+                    "markdown": html_to_markdown(page_html),
+                    "path_info": path_info,
+                    "links_found": list(new_links),
+                }
+
+                # Save individual page
+                self.save_page(url, page_data)
+
+                await asyncio.sleep(DELAY_BETWEEN_REQUESTS)
+                return page_data
+
+            except Exception as e:
+                print(f"Error crawling {url}: {e}")
+                self.failed_urls.add(url)
+                return None
+            finally:
+                if page is not None:
+                    await page.close()
+
+    def save_page(self, url: str, data: dict):
+        """Save page data to a JSON file"""
+        # Create filename from URL path
+        parsed = urlparse(url)
+        path = parsed.path.strip("/") or "index"
+        filename = path.replace("/", "_") + ".json"
+
+        filepath = OUTPUT_DIR / filename
+        with open(filepath, "w", encoding="utf-8") as f:
+            json.dump(data, f, indent=2, ensure_ascii=False)
+
+    async def crawl_all(self):
+        """Main crawl loop"""
+        OUTPUT_DIR.mkdir(exist_ok=True)
+
+        # Start with the docs URL and key sections based on typical CUA docs structure
+        seed_urls = [
+            DOCS_URL,
+            f"{DOCS_URL}/cua",
+            f"{DOCS_URL}/cua/guide",
+            f"{DOCS_URL}/cua/guide/get-started",
+            f"{DOCS_URL}/cua/reference",
+            f"{DOCS_URL}/cua/reference/computer-sdk",
+            f"{DOCS_URL}/cua-bench",
+            f"{BASE_URL}/llms.txt",  # LLM-optimized content if available
+        ]
+
+        for url in seed_urls:
+            normalized = self.normalize_url(url)
+            if self.is_valid_url(normalized) or url.endswith("llms.txt"):
+                self.to_visit.add(normalized)
+
+        async with async_playwright() as playwright:
+            browser = await playwright.chromium.launch(headless=True)
+            try:
+                while self.to_visit:
+                    # Get batch of URLs to crawl
+                    batch = []
+                    while self.to_visit and len(batch) < MAX_CONCURRENT:
+                        url = self.to_visit.pop()
+                        if url not in self.visited_urls:
+                            batch.append(url)
+                            self.visited_urls.add(url)
+
+                    if not batch:
+                        break
+
+                    # Crawl batch concurrently
+                    tasks = [self.crawl_page(browser, url) for url in batch]
+                    results = await asyncio.gather(*tasks)
+
+                    # Collect successful results
+                    for result in results:
+                        if result:
+                            self.all_data.append(result)
+
+                    print(
+                        f"Progress: {len(self.visited_urls)} crawled, "
+                        f"{len(self.to_visit)} remaining"
+                    )
+            finally:
+                await browser.close()
+
+        # Save summary
+        summary = {
+            "total_pages": len(self.all_data),
+            "failed_urls": list(self.failed_urls),
+            "all_urls": list(self.visited_urls),
+            "categories": self._get_categories(),
+        }
+
+        with open(OUTPUT_DIR / "_summary.json", "w", encoding="utf-8") as f:
+            json.dump(summary, f, indent=2)
+
+        # Save all data in one file too
+        with open(OUTPUT_DIR / "_all_pages.json", "w", encoding="utf-8") as f:
+            json.dump(self.all_data, f, indent=2, ensure_ascii=False)
+
+        print("\nCrawl complete!")
+        print(f"Total pages crawled: {len(self.all_data)}")
+        print(f"Failed URLs: {len(self.failed_urls)}")
+        print(f"Output saved to: {OUTPUT_DIR.absolute()}")
+
+    def _get_categories(self) -> dict:
+        """Get summary of categories crawled"""
+        categories = {}
+        for page in self.all_data:
+            cat = page.get("path_info", {}).get("category", "unknown")
+            categories[cat] = categories.get(cat, 0) + 1
+        return categories
+
+
+async def main():
+    crawler = CuaDocsCrawler()
+    await crawler.crawl_all()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

docs/scripts/docs-mcp-server/Dockerfile

@@ -0,0 +1,65 @@
+# syntax=docker/dockerfile:1
+FROM python:3.12-slim AS builder
+
+# Install uv
+COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
+
+# Set working directory
+WORKDIR /app
+
+# Copy project files
+COPY pyproject.toml ./
+
+# Create virtual environment and install dependencies
+RUN uv venv /app/.venv
+ENV VIRTUAL_ENV=/app/.venv
+ENV PATH="/app/.venv/bin:$PATH"
+
+# Install dependencies (without the project itself for better caching)
+RUN uv pip install --no-cache -r pyproject.toml
+
+# Copy application code
+COPY main.py ./
+
+# Production image
+FROM python:3.12-slim AS runtime
+
+# Create non-root user for security
+RUN useradd --create-home --shell /bin/bash appuser
+
+WORKDIR /app
+
+# Copy virtual environment from builder
+COPY --from=builder /app/.venv /app/.venv
+COPY --from=builder /app/main.py /app/main.py
+
+# Set environment variables
+ENV VIRTUAL_ENV=/app/.venv
+ENV PATH="/app/.venv/bin:$PATH"
+ENV PYTHONUNBUFFERED=1
+ENV PORT=8000
+ENV HOST=0.0.0.0
+
+# Database paths (mount volumes to these paths)
+ENV DOCS_DB_PATH=/data/docs_db
+ENV CODE_DB_PATH=/data/code_db
+
+# OpenTelemetry configuration
+ENV OTEL_ENDPOINT=https://otel.cua.ai
+ENV OTEL_SERVICE_NAME=cua-docs-mcp
+
+# Create data directory
+RUN mkdir -p /data && chown appuser:appuser /data
+
+# Switch to non-root user
+USER appuser
+
+# Expose port
+EXPOSE 8000
+
+# Health check
+HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
+    CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1
+
+# Run the server
+CMD ["python", "main.py"]

docs/scripts/docs-mcp-server/main.py

@@ -0,0 +1,521 @@
+"""
+CUA Documentation and Code MCP Server
+
+A standalone MCP server that provides read-only query access to:
+1. CUA documentation (crawled from cua.ai/docs)
+2. Versioned source code indexed across git tags
+
+This server is designed to run as a containerized service, with databases
+mounted from external volumes or cloud storage.
+
+Usage:
+    # Run the server
+    python main.py
+
+    # Or with uvicorn
+    uvicorn main:app --host 0.0.0.0 --port 8000
+"""
+
+import os
+import sqlite3
+import time
+from pathlib import Path
+from typing import Optional
+
+import lancedb
+from fastmcp import FastMCP
+from lancedb.embeddings import get_registry
+from starlette.middleware import Middleware
+from starlette.middleware.cors import CORSMiddleware
+
+# Configuration from environment variables
+OTEL_ENDPOINT = os.environ.get("OTEL_ENDPOINT", "https://otel.cua.ai")
+OTEL_SERVICE_NAME = os.environ.get("OTEL_SERVICE_NAME", "cua-docs-mcp")
+
+# Database paths (configurable via environment)
+DOCS_DB_PATH = os.environ.get("DOCS_DB_PATH", "/data/docs_db")
+CODE_DB_PATH = os.environ.get("CODE_DB_PATH", "/data/code_db")
+
+# Initialize OpenTelemetry for metrics and tracing
+_tracer = None
+_meter = None
+_request_counter = None
+_request_duration = None
+
+
+def init_telemetry():
+    """Initialize OpenTelemetry for metrics and tracing."""
+    global _tracer, _meter, _request_counter, _request_duration
+
+    try:
+        from opentelemetry import metrics, trace
+        from opentelemetry.exporter.otlp.proto.http.metric_exporter import (
+            OTLPMetricExporter,
+        )
+        from opentelemetry.exporter.otlp.proto.http.trace_exporter import (
+            OTLPSpanExporter,
+        )
+        from opentelemetry.sdk.metrics import MeterProvider
+        from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
+        from opentelemetry.sdk.resources import Resource
+        from opentelemetry.sdk.trace import TracerProvider
+        from opentelemetry.sdk.trace.export import BatchSpanProcessor
+
+        resource = Resource.create(
+            {
+                "service.name": OTEL_SERVICE_NAME,
+                "service.version": "1.0.0",
+            }
+        )
+
+        # Set up tracing
+        trace_exporter = OTLPSpanExporter(endpoint=f"{OTEL_ENDPOINT}/v1/traces")
+        tracer_provider = TracerProvider(resource=resource)
+        tracer_provider.add_span_processor(BatchSpanProcessor(trace_exporter))
+        trace.set_tracer_provider(tracer_provider)
+        _tracer = trace.get_tracer(OTEL_SERVICE_NAME)
+
+        # Set up metrics
+        metric_exporter = OTLPMetricExporter(endpoint=f"{OTEL_ENDPOINT}/v1/metrics")
+        metric_reader = PeriodicExportingMetricReader(metric_exporter, export_interval_millis=60000)
+        meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
+        metrics.set_meter_provider(meter_provider)
+        _meter = metrics.get_meter(OTEL_SERVICE_NAME)
+
+        # Create metrics instruments
+        _request_counter = _meter.create_counter(
+            name="mcp_requests_total",
+            description="Total number of MCP tool requests",
+            unit="1",
+        )
+        _request_duration = _meter.create_histogram(
+            name="mcp_request_duration_seconds",
+            description="Duration of MCP tool requests in seconds",
+            unit="s",
+        )
+
+        print(f"OpenTelemetry initialized with endpoint: {OTEL_ENDPOINT}")
+    except ImportError as e:
+        print(f"OpenTelemetry packages not available: {e}")
+    except Exception as e:
+        print(f"Failed to initialize OpenTelemetry: {e}")
+
+
+def record_request(tool_name: str, duration: float, status: str = "success"):
+    """Record metrics for a tool request."""
+    if _request_counter is not None:
+        _request_counter.add(1, {"tool": tool_name, "status": status})
+    if _request_duration is not None:
+        _request_duration.record(duration, {"tool": tool_name, "status": status})
+
+
+# Initialize telemetry
+init_telemetry()
+
+# Initialize the MCP server
+mcp = FastMCP(
+    name="CUA Docs & Code",
+    instructions="""CUA Documentation and Code Server - provides direct read-only query access to Computer Use Agent (CUA) documentation and versioned source code.
+
+=== AVAILABLE TOOLS ===
+
+Documentation:
+- query_docs_db: Execute SQL queries against the documentation SQLite database
+- query_docs_vectors: Execute vector similarity searches against the documentation LanceDB
+
+Code:
+- query_code_db: Execute SQL queries against the code search SQLite database
+- query_code_vectors: Execute vector similarity searches against the code LanceDB
+
+All tools are READ-ONLY. Only SELECT queries are allowed for SQL databases.
+
+=== DOCUMENTATION DATABASE ===
+
+The documentation database contains crawled pages from cua.ai/docs covering:
+- CUA SDK: Python library for building computer-use agents
+- CUA Bench: Benchmarking framework for evaluating computer-use agents
+- Agent Loop: Core execution loop for autonomous agent operation
+- Sandboxes: Docker and cloud VM environments for safe agent execution
+- Computer interfaces: Screen, mouse, keyboard, and bash interaction APIs
+
+=== CODE DATABASE ===
+
+The code database contains versioned source code indexed across all git tags.
+Components include: agent, computer, mcp-server, som, etc.
+
+=== WORKFLOW EXAMPLES ===
+
+1. Find documentation about a topic:
+   - Use query_docs_vectors with a natural language query for semantic search
+   - Use query_docs_db with FTS5 MATCH for keyword search
+
+2. Explore code across versions:
+   - List components: SELECT component, COUNT(DISTINCT version) FROM code_files GROUP BY component
+   - Search code: Use query_code_db with FTS5 on code_files_fts
+   - Get file content: SELECT content FROM code_files WHERE component='agent' AND version='0.7.3' AND file_path='...'
+
+3. Semantic code search:
+   - Use query_code_vectors with natural language queries like "screenshot capture implementation"
+
+IMPORTANT: Always cite sources - URLs for docs, component@version:path for code.""",
+)
+
+# Initialize embedding model - load eagerly to avoid cold start on first search
+print("Initializing embedding model...")
+model = get_registry().get("sentence-transformers").create(name="all-MiniLM-L6-v2")
+
+# Eagerly initialize database connections at startup to reduce first-request latency
+print("Initializing database connections...")
+
+# Docs LanceDB
+_docs_lance_db = None
+_docs_lance_table = None
+docs_db_path = Path(DOCS_DB_PATH)
+if docs_db_path.exists():
+    try:
+        _docs_lance_db = lancedb.connect(docs_db_path)
+        _docs_lance_table = _docs_lance_db.open_table("docs")
+        print(f"  Docs LanceDB loaded from {docs_db_path}")
+    except Exception as e:
+        print(f"  Warning: Could not load docs LanceDB: {e}")
+
+# Docs SQLite
+_docs_sqlite_conn = None
+sqlite_path = Path(DOCS_DB_PATH) / "docs.sqlite"
+if sqlite_path.exists():
+    try:
+        _docs_sqlite_conn = sqlite3.connect(f"file:{sqlite_path}?mode=ro", uri=True)
+        _docs_sqlite_conn.row_factory = sqlite3.Row
+        print(f"  Docs SQLite loaded from {sqlite_path}")
+    except Exception as e:
+        print(f"  Warning: Could not load docs SQLite: {e}")
+
+# Code LanceDB
+_code_lance_db = None
+_code_lance_table = None
+code_lance_path = Path(CODE_DB_PATH) / "code_index.lancedb"
+if code_lance_path.exists():
+    try:
+        _code_lance_db = lancedb.connect(code_lance_path)
+        _code_lance_table = _code_lance_db.open_table("code")
+        print(f"  Code LanceDB loaded from {code_lance_path}")
+    except Exception as e:
+        print(f"  Warning: Could not load code LanceDB: {e}")
+
+# Code SQLite
+_code_sqlite_conn = None
+code_sqlite_path = Path(CODE_DB_PATH) / "code_index.sqlite"
+if code_sqlite_path.exists():
+    try:
+        _code_sqlite_conn = sqlite3.connect(f"file:{code_sqlite_path}?mode=ro", uri=True)
+        _code_sqlite_conn.row_factory = sqlite3.Row
+        print(f"  Code SQLite loaded from {code_sqlite_path}")
+    except Exception as e:
+        print(f"  Warning: Could not load code SQLite: {e}")
+
+print("Database initialization complete.")
+
+
+def get_lance_table():
+    """Get LanceDB connection for docs (eagerly loaded)"""
+    if _docs_lance_table is None:
+        raise RuntimeError(
+            "Database not found. Ensure the docs database is mounted at DOCS_DB_PATH."
+        )
+    return _docs_lance_table
+
+
+def get_sqlite_conn():
+    """Get read-only SQLite connection for docs (eagerly loaded)"""
+    if _docs_sqlite_conn is None:
+        raise RuntimeError(
+            "SQLite database not found. Ensure docs.sqlite is present in DOCS_DB_PATH."
+        )
+    return _docs_sqlite_conn
+
+
+def get_code_lance_table():
+    """Get LanceDB connection for the aggregated code database (eagerly loaded)."""
+    if _code_lance_table is None:
+        raise RuntimeError(
+            "Code LanceDB not found. Ensure code_index.lancedb is present in CODE_DB_PATH."
+        )
+    return _code_lance_table
+
+
+def get_code_sqlite_conn():
+    """Get read-only SQLite connection for the aggregated code database (eagerly loaded)."""
+    if _code_sqlite_conn is None:
+        raise RuntimeError(
+            "Code SQLite database not found. Ensure code_index.sqlite is present in CODE_DB_PATH."
+        )
+    return _code_sqlite_conn
+
+
+# =================== DOCUMENTATION QUERY TOOLS (READ-ONLY) ===================
+
+
+@mcp.tool()
+def query_docs_db(sql: str) -> list[dict]:
+    """
+    Execute a SQL query against the documentation database.
+    The database is READ-ONLY.
+
+    Database Schema:
+
+    Table: pages
+    - id INTEGER PRIMARY KEY AUTOINCREMENT
+    - url TEXT NOT NULL UNIQUE         -- Full URL of the documentation page
+    - title TEXT NOT NULL              -- Page title
+    - category TEXT NOT NULL           -- Category (e.g., 'cua', 'cuabench', 'llms.txt')
+    - content TEXT NOT NULL            -- Plain text content (markdown stripped)
+
+    Virtual Table: pages_fts (FTS5 full-text search)
+    - content TEXT                     -- Full-text indexed content
+    - url TEXT UNINDEXED
+    - title TEXT UNINDEXED
+    - category TEXT UNINDEXED
+
+    Example queries:
+
+    1. List all pages: SELECT url, title, category FROM pages ORDER BY category, title
+
+    2. Full-text search with snippets:
+       SELECT p.url, p.title, snippet(pages_fts, 0, '>>>', '<<<', '...', 64) as snippet
+       FROM pages_fts JOIN pages p ON pages_fts.rowid = p.id
+       WHERE pages_fts MATCH 'agent loop' ORDER BY rank LIMIT 10
+
+    3. Get page content: SELECT url, title, content FROM pages WHERE url LIKE '%quickstart%'
+
+    Args:
+        sql: SQL query to execute
+
+    Returns:
+        List of dictionaries, one per row, with column names as keys
+    """
+    start_time = time.perf_counter()
+    status = "success"
+    try:
+        conn = get_sqlite_conn()
+        cursor = conn.cursor()
+        cursor.execute(sql)
+        return [dict(row) for row in cursor.fetchall()]
+    except Exception:
+        status = "error"
+        raise
+    finally:
+        record_request("query_docs_db", time.perf_counter() - start_time, status)
+
+
+@mcp.tool()
+def query_docs_vectors(
+    query: str,
+    limit: int = 10,
+    where: Optional[str] = None,
+    select: Optional[list[str]] = None,
+) -> list[dict]:
+    """
+    Execute a vector similarity search against the documentation LanceDB (read-only).
+
+    Schema:
+    - text TEXT           -- The document chunk text
+    - vector VECTOR       -- Embedding vector (all-MiniLM-L6-v2, 384 dimensions)
+    - url TEXT            -- Source URL
+    - title TEXT          -- Document title
+    - category TEXT       -- Category (e.g., 'cua', 'cuabench')
+    - chunk_index INT     -- Index of chunk within document
+
+    Args:
+        query: Natural language query to embed and search for
+        limit: Maximum number of results (default: 10, max: 100)
+        where: Optional SQL-like filter (e.g., "category = 'cua'")
+        select: Optional list of columns to return (default: all except vector)
+
+    Returns:
+        List of matching documents with similarity scores (_distance field)
+    """
+    start_time = time.perf_counter()
+    status = "success"
+    try:
+        limit = min(max(1, limit), 100)
+        table = get_lance_table()
+
+        search = table.search(query).limit(limit)
+
+        if where:
+            search = search.where(where)
+        if select:
+            search = search.select(select)
+
+        results = search.to_list()
+
+        formatted = []
+        for r in results:
+            result = {}
+            for key, value in r.items():
+                if key == "vector":
+                    continue
+                result[key] = value
+            formatted.append(result)
+
+        return formatted
+    except Exception:
+        status = "error"
+        raise
+    finally:
+        record_request("query_docs_vectors", time.perf_counter() - start_time, status)
+
+
+# =================== CODE QUERY TOOLS (READ-ONLY) ===================
+
+
+@mcp.tool()
+def query_code_db(sql: str) -> list[dict]:
+    """
+    Execute a SQL query against the code search database.
+    The database is READ-ONLY.
+
+    Database Schema:
+
+    Table: code_files
+    - id INTEGER PRIMARY KEY AUTOINCREMENT
+    - component TEXT NOT NULL          -- Component name (e.g., "agent", "computer")
+    - version TEXT NOT NULL            -- Version string (e.g., "0.7.3")
+    - file_path TEXT NOT NULL          -- Path to file
+    - content TEXT NOT NULL            -- Full source code content
+    - language TEXT NOT NULL           -- Programming language
+    - UNIQUE(component, version, file_path)
+
+    Virtual Table: code_files_fts (FTS5 full-text search)
+    - content TEXT                     -- Full-text indexed content
+    - component TEXT UNINDEXED
+    - version TEXT UNINDEXED
+    - file_path TEXT UNINDEXED
+
+    Example queries:
+
+    1. List components: SELECT component, COUNT(DISTINCT version) as version_count
+       FROM code_files GROUP BY component ORDER BY component
+
+    2. List versions: SELECT DISTINCT version FROM code_files
+       WHERE component = 'agent' ORDER BY version DESC
+
+    3. Full-text search:
+       SELECT f.component, f.version, f.file_path,
+              snippet(code_files_fts, 0, '>>>', '<<<', '...', 64) as snippet
+       FROM code_files_fts JOIN code_files f ON code_files_fts.rowid = f.id
+       WHERE code_files_fts MATCH 'ComputerAgent' ORDER BY rank LIMIT 10
+
+    4. Get file content: SELECT content, language FROM code_files
+       WHERE component = 'agent' AND version = '0.7.3' AND file_path = 'agent/core.py'
+
+    Args:
+        sql: SQL query to execute
+
+    Returns:
+        List of dictionaries, one per row, with column names as keys
+    """
+    start_time = time.perf_counter()
+    status = "success"
+    try:
+        conn = get_code_sqlite_conn()
+        cursor = conn.cursor()
+        cursor.execute(sql)
+        return [dict(row) for row in cursor.fetchall()]
+    except Exception:
+        status = "error"
+        raise
+    finally:
+        record_request("query_code_db", time.perf_counter() - start_time, status)
+
+
+@mcp.tool()
+def query_code_vectors(
+    query: str,
+    limit: int = 10,
+    where: Optional[str] = None,
+    select: Optional[list[str]] = None,
+    component: Optional[str] = None,
+) -> list[dict]:
+    """
+    Execute a vector similarity search against the code LanceDB (read-only).
+
+    Schema:
+    - text TEXT           -- The source code content
+    - vector VECTOR       -- Embedding vector (all-MiniLM-L6-v2, 384 dimensions)
+    - component TEXT      -- Component name (e.g., "agent", "computer")
+    - version TEXT        -- Version string (e.g., "0.7.3")
+    - file_path TEXT      -- Path to file within the component
+    - language TEXT       -- Programming language
+
+    Args:
+        query: Natural language query to embed and search for
+        limit: Maximum number of results (default: 10, max: 100)
+        where: Optional SQL-like filter (e.g., "version = '0.7.3'")
+        select: Optional list of columns to return (default: all except vector)
+        component: Optional component to filter by (if not specified, searches all)
+
+    Returns:
+        List of matching code files with similarity scores (_distance field)
+    """
+    start_time = time.perf_counter()
+    status = "success"
+    try:
+        limit = min(max(1, limit), 100)
+        table = get_code_lance_table()
+
+        search = table.search(query).limit(limit)
+
+        # Build where clause, adding component filter if specified
+        where_clauses = []
+        if component:
+            where_clauses.append(f"component = '{component}'")
+        if where:
+            where_clauses.append(where)
+
+        if where_clauses:
+            search = search.where(" AND ".join(where_clauses))
+        if select:
+            search = search.select(select)
+
+        results = search.to_list()
+
+        formatted = []
+        for r in results:
+            result = {}
+            for key, value in r.items():
+                if key == "vector":
+                    continue
+                result[key] = value
+            formatted.append(result)
+
+        return formatted
+    except Exception:
+        status = "error"
+        raise
+    finally:
+        record_request("query_code_vectors", time.perf_counter() - start_time, status)
+
+
+# Create the ASGI app
+app = mcp.http_app(
+    transport="streamable-http",
+    middleware=[
+        Middleware(
+            CORSMiddleware,
+            allow_origins=["*"],
+            allow_credentials=True,
+            allow_methods=["*"],
+            allow_headers=["*"],
+        )
+    ],
+)
+
+if __name__ == "__main__":
+    import uvicorn
+
+    port = int(os.environ.get("PORT", "8000"))
+    host = os.environ.get("HOST", "0.0.0.0")
+
+    print(f"Starting MCP server on {host}:{port}")
+    uvicorn.run(app, host=host, port=port)

docs/scripts/docs-mcp-server/pyproject.toml

@@ -0,0 +1,30 @@
+[project]
+name = "docs-mcp-server"
+description = "MCP Server for CUA Documentation and Code Search"
+version = "0.1.0"
+requires-python = ">=3.12,<3.14"
+authors = [
+    {name = "TryCua", email = "gh@trycua.com"}
+]
+dependencies = [
+    "fastmcp>=2.14.0",
+    "lancedb>=0.4.0",
+    "sentence-transformers>=2.2.0",
+    "pyarrow>=14.0.1",
+    "pydantic>=2.0.0",
+    "pandas>=2.0.0",
+    "markdown-it-py>=3.0.0",
+    "opentelemetry-api>=1.20.0",
+    "opentelemetry-sdk>=1.20.0",
+    "opentelemetry-exporter-otlp-proto-http>=1.20.0",
+    "uvicorn>=0.23.0",
+]
+
+[project.scripts]
+docs-mcp-server = "main:app"
+
+[tool.uv]
+dev-dependencies = [
+    "black>=23.9.1",
+    "ruff>=0.0.292",
+]

docs/scripts/generate_db.py

@@ -0,0 +1,261 @@
+"""
+Database generator for CUA documentation
+Parses crawled JSON data and creates a LanceDB vector database for RAG
+"""
+
+import json
+import re
+from pathlib import Path
+from typing import Optional
+
+import lancedb
+from lancedb.embeddings import get_registry
+from lancedb.pydantic import LanceModel, Vector
+
+# Configuration
+CRAWLED_DATA_DIR = Path(__file__).parent.parent / "crawled_data"
+DB_PATH = Path(__file__).parent.parent / "docs_db"
+CHUNK_SIZE = 1000  # Characters per chunk
+CHUNK_OVERLAP = 200  # Overlap between chunks
+
+# Use sentence-transformers for embeddings
+model = get_registry().get("sentence-transformers").create(name="all-MiniLM-L6-v2")
+
+
+class DocChunk(LanceModel):
+    """Schema for document chunks in the database"""
+
+    text: str = model.SourceField()
+    vector: Vector(model.ndims()) = model.VectorField()
+    url: str
+    title: str
+    category: str
+    subcategory: Optional[str]
+    page: str
+    chunk_index: int
+
+
+def clean_markdown(markdown: str) -> str:
+    """Clean markdown content for better chunking"""
+    # Remove excessive whitespace
+    text = re.sub(r"\n{3,}", "\n\n", markdown)
+    # Remove image markdown
+    text = re.sub(r"!\[.*?\]\(.*?\)", "", text)
+    # Remove link URLs but keep text
+    text = re.sub(r"\[([^\]]+)\]\([^)]+\)", r"\1", text)
+    # Remove HTML tags
+    text = re.sub(r"<[^>]+>", "", text)
+    # Clean up whitespace
+    text = re.sub(r" {2,}", " ", text)
+    return text.strip()
+
+
+def chunk_text(text: str, chunk_size: int = CHUNK_SIZE, overlap: int = CHUNK_OVERLAP) -> list[str]:
+    """Split text into overlapping chunks, respecting sentence boundaries"""
+    if not text:
+        return []
+
+    # Split by paragraphs first
+    paragraphs = text.split("\n\n")
+    chunks = []
+    current_chunk = ""
+
+    for para in paragraphs:
+        para = para.strip()
+        if not para:
+            continue
+
+        # If adding this paragraph exceeds chunk size, save current and start new
+        if len(current_chunk) + len(para) + 2 > chunk_size:
+            if current_chunk:
+                chunks.append(current_chunk.strip())
+                # Start new chunk with overlap from previous
+                if overlap > 0 and len(current_chunk) > overlap:
+                    # Try to find a sentence boundary for overlap
+                    overlap_text = current_chunk[-overlap:]
+                    sentence_end = overlap_text.rfind(". ")
+                    if sentence_end > 0:
+                        overlap_text = overlap_text[sentence_end + 2 :]
+                    current_chunk = overlap_text + "\n\n" + para
+                else:
+                    current_chunk = para
+            else:
+                # Single paragraph exceeds chunk size, split by sentences
+                sentences = re.split(r"(?<=[.!?])\s+", para)
+                for sentence in sentences:
+                    if len(current_chunk) + len(sentence) + 1 > chunk_size:
+                        if current_chunk:
+                            chunks.append(current_chunk.strip())
+                            # Start new chunk with overlap from previous, similar to paragraph logic
+                            if overlap > 0 and len(current_chunk) > overlap:
+                                overlap_text = current_chunk[-overlap:]
+                                sentence_end = overlap_text.rfind(". ")
+                                if sentence_end > 0:
+                                    overlap_text = overlap_text[sentence_end + 2 :]
+                                current_chunk = (overlap_text + " " + sentence).strip()
+                            else:
+                                current_chunk = sentence.strip()
+                        else:
+                            # No existing chunk; start with this sentence
+                            current_chunk = sentence.strip()
+                    else:
+                        current_chunk = (current_chunk + " " + sentence).strip()
+        else:
+            current_chunk = (current_chunk + "\n\n" + para).strip()
+
+    # Don't forget the last chunk
+    if current_chunk:
+        chunks.append(current_chunk.strip())
+
+    return chunks
+
+
+def load_crawled_data() -> list[dict]:
+    """Load all crawled page data"""
+    all_pages_file = CRAWLED_DATA_DIR / "_all_pages.json"
+
+    if all_pages_file.exists():
+        with open(all_pages_file, "r", encoding="utf-8") as f:
+            return json.load(f)
+
+    # Fallback: load individual files
+    pages = []
+    for json_file in CRAWLED_DATA_DIR.glob("*.json"):
+        if json_file.name.startswith("_"):
+            continue
+        with open(json_file, "r", encoding="utf-8") as f:
+            pages.append(json.load(f))
+
+    return pages
+
+
+def process_pages(pages: list[dict]) -> list[dict]:
+    """Process pages into document chunks"""
+    all_chunks = []
+
+    for page in pages:
+        markdown = page.get("markdown", "")
+        if not markdown:
+            continue
+
+        # Clean the markdown
+        cleaned_text = clean_markdown(markdown)
+        if not cleaned_text or len(cleaned_text) < 50:
+            continue
+
+        # Get path info
+        path_info = page.get("path_info", {})
+
+        # Chunk the text
+        text_chunks = chunk_text(cleaned_text)
+
+        # Ensure non-null values for required fields
+        url = page.get("url", "")
+        title = page.get("title") or path_info.get("page", "") or "Untitled"
+        category = path_info.get("category") or "unknown"
+        page_name = path_info.get("page") or ""
+
+        for i, chunk_text_content in enumerate(text_chunks):
+            chunk = {
+                "text": chunk_text_content,
+                "url": url,
+                "title": title,
+                "category": category,
+                "subcategory": path_info.get("subcategory"),
+                "page": page_name,
+                "chunk_index": i,
+            }
+            all_chunks.append(chunk)
+
+    return all_chunks
+
+
+def create_database(chunks: list[dict]):
+    """Create LanceDB database from chunks"""
+    # Remove existing database
+    if DB_PATH.exists():
+        import shutil
+
+        shutil.rmtree(DB_PATH)
+
+    # Create database
+    db = lancedb.connect(DB_PATH)
+
+    # Create table with schema
+    table = db.create_table(
+        "docs",
+        schema=DocChunk,
+        mode="overwrite",
+    )
+
+    # Add data in batches
+    batch_size = 100
+    for i in range(0, len(chunks), batch_size):
+        batch = chunks[i : i + batch_size]
+        print(f"Adding batch {i // batch_size + 1}/{(len(chunks) + batch_size - 1) // batch_size}")
+        table.add(batch)
+
+    print(f"Database created at: {DB_PATH}")
+    print(f"Total chunks: {len(chunks)}")
+
+    return db
+
+
+def test_search(db: lancedb.DBConnection, query: str, limit: int = 5):
+    """Test search functionality"""
+    table = db.open_table("docs")
+
+    print(f"\nSearching for: '{query}'")
+    print("-" * 50)
+
+    results = table.search(query).limit(limit).to_list()
+
+    for i, result in enumerate(results):
+        print(f"\n{i + 1}. [{result['category']}] {result['title']}")
+        print(f"   URL: {result['url']}")
+        print(f"   Score: {result.get('_distance', 'N/A'):.4f}")
+        print(f"   Preview: {result['text'][:150]}...")
+
+
+def main():
+    print("Loading crawled data...")
+    pages = load_crawled_data()
+    print(f"Loaded {len(pages)} pages")
+
+    if not pages:
+        print("No crawled data found. Run crawl_docs.py first.")
+        return
+
+    print("\nProcessing pages into chunks...")
+    chunks = process_pages(pages)
+    print(f"Created {len(chunks)} chunks")
+
+    if not chunks:
+        print("No chunks created. Check your crawled data.")
+        return
+
+    print("\nCreating database...")
+    db = create_database(chunks)
+
+    # Test with sample queries
+    print("\n" + "=" * 50)
+    print("Testing search functionality")
+    print("=" * 50)
+
+    test_queries = [
+        "how to install CUA",
+        "computer use agent",
+        "benchmark evaluation",
+        "API reference",
+    ]
+
+    for query in test_queries:
+        test_search(db, query)
+
+    print("\n" + "=" * 50)
+    print("Database generation complete!")
+    print(f"Database location: {DB_PATH}")
+
+
+if __name__ == "__main__":
+    main()

docs/scripts/generate_sqlite.py

@@ -0,0 +1,278 @@
+"""
+SQLite database generator for CUA documentation
+Creates a full-text search enabled SQLite database from crawled data
+"""
+
+import json
+import re
+import sqlite3
+from pathlib import Path
+
+from markdown_it import MarkdownIt
+
+# Configuration
+CRAWLED_DATA_DIR = Path(__file__).parent.parent / "crawled_data"
+SQLITE_PATH = Path(__file__).parent.parent / "docs_db" / "docs.sqlite"
+
+
+def clean_markdown(markdown: str) -> str:
+    """
+    Extract plain text content from markdown using a proper markdown parser.
+
+    This function uses markdown-it-py to parse the markdown into a token tree
+    and then extracts only the text content, removing:
+    - Markdown formatting (bold, italic, headers, etc.)
+    - Links (keeping only the link text)
+    - Images (alt text is discarded)
+    - HTML tags
+    - Code block language identifiers
+
+    Args:
+        markdown: Raw markdown content
+
+    Returns:
+        Plain text content suitable for full-text search
+    """
+    md_parser = MarkdownIt()
+    tokens = md_parser.parse(markdown)
+
+    text_parts = []
+
+    def extract_text(token_list):
+        """Recursively extract text from token tree"""
+        for token in token_list:
+            if token.type == "inline" and token.children:
+                # Process inline content (text, links, formatting, etc.)
+                for child in token.children:
+                    if child.type == "text":
+                        text_parts.append(child.content)
+                    elif child.type == "code_inline":
+                        text_parts.append(child.content)
+                    elif child.type == "softbreak":
+                        text_parts.append(" ")
+                    elif child.type == "hardbreak":
+                        text_parts.append("\n")
+                    # Skip link markup, images, and formatting tokens
+                    # (link_open, link_close, image, strong_open, strong_close, em_open, em_close, etc.)
+            elif token.type == "fence" or token.type == "code_block":
+                # Include code content and add newline after
+                text_parts.append(token.content)
+                text_parts.append("\n")
+            elif token.type == "html_block" or token.type == "html_inline":
+                # Skip HTML blocks and inline HTML
+                pass
+
+            # Recursively process nested children
+            if token.children:
+                extract_text(token.children)
+
+            # Add spacing after block elements
+            if token.type in [
+                "heading_close",
+                "paragraph_close",
+                "list_item_close",
+                "blockquote_close",
+            ]:
+                text_parts.append("\n")
+
+    extract_text(tokens)
+
+    # Join and clean up whitespace
+    text = "".join(text_parts)
+    # Normalize multiple newlines to at most double newlines
+    text = re.sub(r"\n{3,}", "\n\n", text)
+    # Normalize multiple spaces to single space within lines
+    text = re.sub(r" {2,}", " ", text)
+
+    return text.strip()
+
+
+def load_crawled_data() -> list[dict]:
+    """Load all crawled page data"""
+    all_pages_file = CRAWLED_DATA_DIR / "_all_pages.json"
+
+    if all_pages_file.exists():
+        with open(all_pages_file, "r", encoding="utf-8") as f:
+            return json.load(f)
+
+    pages = []
+    for json_file in CRAWLED_DATA_DIR.glob("*.json"):
+        if json_file.name.startswith("_"):
+            continue
+        with open(json_file, "r", encoding="utf-8") as f:
+            pages.append(json.load(f))
+
+    return pages
+
+
+def create_database(pages: list[dict]):
+    """Create SQLite database with FTS5 full-text search"""
+    # Ensure parent directory exists
+    SQLITE_PATH.parent.mkdir(parents=True, exist_ok=True)
+
+    # Remove existing database
+    if SQLITE_PATH.exists():
+        SQLITE_PATH.unlink()
+
+    conn = sqlite3.connect(SQLITE_PATH)
+    cursor = conn.cursor()
+
+    # Create main pages table
+    cursor.execute(
+        """
+        CREATE TABLE pages (
+            id INTEGER PRIMARY KEY AUTOINCREMENT,
+            url TEXT UNIQUE NOT NULL,
+            title TEXT,
+            description TEXT,
+            category TEXT,
+            subcategory TEXT,
+            page_name TEXT,
+            content TEXT,
+            raw_markdown TEXT
+        )
+    """
+    )
+
+    # Create FTS5 virtual table for full-text search
+    cursor.execute(
+        """
+        CREATE VIRTUAL TABLE pages_fts USING fts5(
+            content,
+            url UNINDEXED,
+            title UNINDEXED,
+            category UNINDEXED,
+            content='pages',
+            content_rowid='id'
+        )
+    """
+    )
+
+    # Create triggers to keep FTS in sync
+    cursor.execute(
+        """
+        CREATE TRIGGER pages_ai AFTER INSERT ON pages BEGIN
+            INSERT INTO pages_fts(rowid, content, url, title, category)
+            VALUES (new.id, new.content, new.url, new.title, new.category);
+        END;
+    """
+    )
+
+    cursor.execute(
+        """
+        CREATE TRIGGER pages_ad AFTER DELETE ON pages BEGIN
+            DELETE FROM pages_fts WHERE rowid = old.id;
+        END;
+    """
+    )
+
+    cursor.execute(
+        """
+        CREATE TRIGGER pages_au AFTER UPDATE ON pages BEGIN
+            DELETE FROM pages_fts WHERE rowid = old.id;
+            INSERT INTO pages_fts(rowid, content, url, title, category)
+            VALUES (new.id, new.content, new.url, new.title, new.category);
+        END;
+    """
+    )
+
+    # Insert pages
+    for page in pages:
+        markdown = page.get("markdown", "")
+        if not markdown:
+            continue
+
+        content = clean_markdown(markdown)
+        if not content or len(content) < 50:
+            continue
+
+        path_info = page.get("path_info", {})
+
+        cursor.execute(
+            """
+            INSERT OR REPLACE INTO pages
+            (url, title, description, category, subcategory, page_name, content, raw_markdown)
+            VALUES (?, ?, ?, ?, ?, ?, ?, ?)
+        """,
+            (
+                page.get("url", ""),
+                page.get("title") or path_info.get("page", "") or "Untitled",
+                page.get("description", ""),
+                path_info.get("category", "unknown"),
+                path_info.get("subcategory"),
+                path_info.get("page", ""),
+                content,
+                markdown,
+            ),
+        )
+
+    conn.commit()
+
+    # Get stats
+    cursor.execute("SELECT COUNT(*) FROM pages")
+    page_count = cursor.fetchone()[0]
+
+    cursor.execute("SELECT category, COUNT(*) FROM pages GROUP BY category")
+    categories = cursor.fetchall()
+
+    conn.close()
+
+    print(f"SQLite database created at: {SQLITE_PATH}")
+    print(f"Total pages: {page_count}")
+    print("Pages by category:")
+    for cat, count in categories:
+        print(f"  - {cat}: {count}")
+
+
+def test_search(query: str):
+    """Test full-text search"""
+    conn = sqlite3.connect(SQLITE_PATH)
+    cursor = conn.cursor()
+
+    print(f"\nFTS5 search for: '{query}'")
+    print("-" * 50)
+
+    cursor.execute(
+        """
+        SELECT url, title, snippet(pages_fts, 0, '>>>', '<<<', '...', 50) as snippet
+        FROM pages_fts
+        WHERE pages_fts MATCH ?
+        ORDER BY rank
+        LIMIT 5
+    """,
+        (query,),
+    )
+
+    results = cursor.fetchall()
+    for url, title, snippet in results:
+        print(f"\n{title}")
+        print(f"  URL: {url}")
+        print(f"  Snippet: {snippet}")
+
+    conn.close()
+
+
+def main():
+    print("Loading crawled data...")
+    pages = load_crawled_data()
+    print(f"Loaded {len(pages)} pages")
+
+    if not pages:
+        print("No crawled data found. Run crawl_docs.py first.")
+        return
+
+    print("\nCreating SQLite database...")
+    create_database(pages)
+
+    # Test searches
+    print("\n" + "=" * 50)
+    print("Testing FTS5 search")
+    print("=" * 50)
+
+    test_search("install")
+    test_search("computer use agent")
+    test_search("benchmark")
+
+
+if __name__ == "__main__":
+    main()