light-heart-labs--dreamserver/ARCHITECTURE.md

ODS Architecture

Version 2.5.3 | Fully local AI stack deployed on user hardware with a single command

Overview

ODS is a self-hosted AI platform built around 24 bundled service manifests, Docker Compose services, and a small number of host-managed helpers across NVIDIA, AMD, Apple Silicon, Intel Arc, and CPU/cloud fallback paths. The system is structured in two layers: an outer wrapper (installer scripts, CI, resources) and the core product (ods/) containing all deployable code.

The architecture follows a layered compose model: a base compose file defines core services, GPU-specific overlays configure hardware acceleration, and extension compose files add optional services. A registry-driven CLI (ods-cli) manages the lifecycle.

System Architecture

graph TB
    subgraph External["User Access (localhost only)"]
        Browser["Browser"]
    end

    subgraph Core["Core Services"]
        LLAMA["llama-server<br/>:8080<br/>LLM Inference"]
        WEBUI["open-webui<br/>:3000<br/>Chat UI"]
        DASH["dashboard<br/>:3001<br/>Control Center"]
        DAPI["dashboard-api<br/>:3002<br/>System Status API"]
    end

    subgraph Gateway["API Gateway"]
        LITE["litellm<br/>:4000<br/>OpenAI-compatible proxy"]
    end

    subgraph Voice["Voice Pipeline"]
        WHISPER["whisper<br/>:9000<br/>Speech-to-Text"]
        TTS["tts / Kokoro<br/>:8880<br/>Text-to-Speech"]
    end

    subgraph Search["Search & Research"]
        SEARX["searxng<br/>:8888<br/>Metasearch"]
        PERP["perplexica<br/>:3004<br/>Deep Research"]
    end

    subgraph Agents["Agents & Automation"]
        HERMES["hermes<br/>:9120 via proxy<br/>Default Agent"]
        CLAW["openclaw<br/>:7860<br/>Deprecated Agent"]
        APE["ape<br/>:7890<br/>Policy Engine"]
        N8N["n8n<br/>:5678<br/>Workflows"]
    end

    subgraph RAG["RAG Pipeline"]
        QDRANT["qdrant<br/>:6333<br/>Vector DB"]
        EMBED["embeddings<br/>:8090<br/>TEI Vectors"]
    end

    subgraph Media["Media Generation"]
        COMFY["comfyui<br/>:8188<br/>Image Gen"]
    end

    subgraph Privacy["Privacy & Observability"]
        SHIELD["privacy-shield<br/>:8085<br/>PII Protection"]
        SPY["token-spy<br/>:3005<br/>Usage Monitor"]
        LANG["langfuse<br/>:3006<br/>LLM Tracing"]
    end

    subgraph Dev["Development"]
        CODE["opencode<br/>:3003<br/>Web IDE"]
    end

    subgraph Access["LAN / Remote Access"]
        PROXY["ods-proxy<br/>:80<br/>mDNS web entry"]
        TAIL["tailscale<br/>host network<br/>remote access"]
    end

    Browser --> WEBUI
    Browser --> DASH
    Browser --> LITE
    Browser --> CODE
    Browser --> HERMES
    Browser --> PROXY

    WEBUI --> LLAMA
    WEBUI --> COMFY
    WEBUI --> WHISPER
    WEBUI --> TTS
    WEBUI --> SEARX

    LITE --> LLAMA

    DASH --> DAPI
    DAPI --> LLAMA
    DAPI --> N8N
    DAPI --> SPY
    DAPI --> SHIELD

    HERMES --> LLAMA
    CLAW --> LLAMA
    CLAW --> SEARX

    PERP --> LLAMA
    PERP --> SEARX

    SHIELD --> LLAMA
    PROXY --> WEBUI
    PROXY --> DASH
    PROXY --> HERMES

Functional Areas

1. Inference Layer

The LLM inference engine (llama-server) is the foundation. GPU overlays select the correct container image and runtime:

Backend	Image	Acceleration
NVIDIA	llama.cpp:server-cuda-b9014 default, overrideable via LLAMA_SERVER_IMAGE	CUDA, all GPUs reserved
AMD	Custom ods-lemonade-server	ROCm / Vulkan / NPU via Lemonade
Apple	Native host llama-server via macOS installer; Docker overlay is CPU fallback	Metal on host; containers reach host.docker.internal:8080
Intel Arc	SYCL backend	Experimental
CPU	llama.cpp:server-b8248	Pure CPU fallback

LiteLLM (port 4000) sits in front as an OpenAI-compatible proxy, enabling cloud fallback in hybrid mode and standardized API access for all consumers.

2. Chat & UI Layer

• open-webui (port 3000) — Primary chat interface with integrated image generation (ComfyUI/SDXL), voice I/O (Whisper + Kokoro), and web search (SearXNG)
• dashboard (port 3001) — React/Vite control center for feature discovery, service health, setup wizard, model management
• dashboard-api (port 3002) — FastAPI backend with routers for setup, features, agents, privacy, workflows, and updates

3. Search & Research

• searxng (port 8888) — Privacy-respecting metasearch engine
• perplexica (port 3004) — Deep research combining search results with LLM reasoning

4. Agents & Automation

• hermes (internal 9119, auth proxy on 9120) — default agent, fronted by hermes-proxy and magic-link auth
• openclaw (port 7860) — deprecated optional agent framework retained for compatibility
• ape (port 7890) — Agent Policy Engine enforcing allow/deny rules on tool access
• n8n (port 5678) — Visual workflow automation with a pre-built catalog

5. RAG Pipeline

• qdrant (port 6333) — Vector database for document retrieval
• embeddings (port 8090) — HuggingFace TEI for generating vector embeddings

6. Voice Pipeline

• whisper (port 9000) — Speech-to-text (OpenAI-compatible API)
• tts/Kokoro (port 8880) — Text-to-speech (OpenAI-compatible API)

7. Media Generation

• comfyui (port 8188) — Image generation with SDXL Lightning (4-step)

8. Privacy & Observability

• privacy-shield (port 8085) — PII detection and scrubbing middleware
• token-spy (port 3005) — Token usage and cost tracking
• langfuse (port 3006) — LLM observability and tracing

9. Development

• opencode (port 3003) — Web IDE (runs as host systemd service, not Docker)

Installer Architecture

The installer is a 13-phase pipeline orchestrated by install-core.sh. Installer libraries in installers/lib/ are pure functions (no side effects); lib/service-registry.sh loads service manifests and port metadata; phases in installers/phases/ execute sequentially.

graph LR
    subgraph Libraries["installers/lib/ (pure functions)"]
        C[constants] --> D[detection]
        D --> T[tier-map]
        T --> CS[compose-select]
        P[packaging]
        PY[python-runtime]
        DI[docker-images]
        PR[progress]
        SR[service-registry]
        U[ui]
        L[logging]
    end

    subgraph Phases["installers/phases/ (sequential)"]
        P01["01 Preflight"] --> P02["02 Detection"]
        P02 --> P03["03 Features"]
        P03 --> P04["04 Requirements"]
        P04 --> P05["05 Docker"]
        P05 --> P06["06 Directories"]
        P06 --> P07["07 DevTools"]
        P07 --> P08["08 Images"]
        P08 --> P09["09 Offline"]
        P09 --> P10["10 AMD Tuning"]
        P10 --> P11["11 Services"]
        P11 --> P12["12 Health"]
        P12 --> P13["13 Summary"]
    end

    Libraries --> Phases

Phase	Purpose
01 Preflight	Root/OS/tools checks, existing install detection
02 Detection	GPU hardware detection, tier assignment, compose config selection
03 Features	Interactive feature selection (voice, workflows, RAG, images, etc.)
04 Requirements	RAM, disk, GPU, port availability checks
05 Docker	Install Docker, Compose, NVIDIA Container Toolkit
06 Directories	Create dirs, copy source, generate .env, configure services
07 DevTools	Install Claude Code, Codex CLI, OpenCode
08 Images	Build image pull list, download all Docker images
09 Offline	Configure air-gapped operation
10 AMD Tuning	AMD APU sysctl, modprobe, GRUB, tuned setup
11 Services	Download GGUF model, generate models.ini, launch stack
12 Health	Verify services responding, configure Perplexica, pre-download STT models
13 Summary	Generate URLs, desktop shortcuts, summary JSON

Generated config is written in more than one place. When changing .env, OpenCode, Perplexica, Hermes, or LiteLLM/Lemonade behavior, review docs/INSTALLER-ARCHITECTURE.md#generated-config-writers before merging.

Docker Compose Layering

The stack uses compose file merging. The resolver script dynamically discovers enabled extensions and composes the full stack:

graph TB
    BASE["docker-compose.base.yml<br/>(core services)"]
    GPU["docker-compose.{nvidia,amd,apple,cpu}.yml<br/>(GPU overlay)"]
    EXT1["extensions/services/comfyui/compose.yaml"]
    EXT2["extensions/services/n8n/compose.yaml"]
    EXT3["extensions/services/.../compose.yaml"]
    EXTGPU["extensions/services/.../compose.nvidia.yaml"]

    BASE --> MERGE["resolve-compose-stack.sh"]
    GPU --> MERGE
    EXT1 --> MERGE
    EXT2 --> MERGE
    EXT3 --> MERGE
    EXTGPU --> MERGE
    MERGE --> STACK["Final Docker Compose Stack"]

Key Execution Flows

1. Installation Flow

install.sh → install-core.sh → sources installers/lib/*.sh → sources installers/phases/01..13.sh sequentially. Each phase reads state set by prior phases via exported variables. Hardware detection (phase 02) drives all downstream decisions: tier assignment selects the model GGUF, context window, batch size, and compose overlays.

2. Service Startup Flow

ods-cli start → resolve-compose-stack.sh reads enabled services from .env → assembles docker compose -f base -f gpu-overlay -f ext1 -f ext2 ... → docker compose up -d. Health checks gate dependent services (e.g., open-webui waits for llama-server healthy).

3. Chat Request Flow

Browser → open-webui:3000 → llama-server:8080/v1/chat/completions → GPU inference → response streamed back. If hybrid mode: open-webui → litellm:4000 → tries llama-server first, falls back to cloud API.

4. Agent Execution Flow

Browser → hermes-proxy:9120 → magic-link auth gate → ods-hermes:9119 inside the Docker network → Hermes tools/search/reasoning → local LLM via an OpenAI-compatible endpoint. OpenClaw still exists as a deprecated optional agent on :7860; APE provides policy/audit controls for agent tool surfaces.

5. Dashboard Feature Discovery Flow

Browser → dashboard:3001 → dashboard-api:3002/api/features → API reads all service manifests, checks container health via Docker socket, cross-references GPU capabilities and VRAM → returns feature list with status (enabled, available, insufficient_vram, services_needed) and recommendations.

Configuration

Environment Variables (Key Connections)

Variable	Default	Controls
GPU_BACKEND	detected	nvidia, amd, apple, cpu
GGUF_FILE	tier-dependent	Model file in /data/models/
CTX_SIZE	16384	Context window (tokens)
ODS_MODE	local	local, cloud, hybrid
LITELLM_KEY	generated	API gateway authentication
DASHBOARD_API_KEY	generated	Dashboard API authentication

Port Map

All services bind to 127.0.0.1 (localhost only). Canonical port assignments live in config/ports.json.

Port	Service	Port	Service
80	ods-proxy	3000	open-webui
3001	dashboard	3002	dashboard-api
3003	opencode	3004	perplexica
3005	token-spy	3006	langfuse
4000	litellm	5678	n8n
6333	qdrant	6334	qdrant gRPC
7860	openclaw	7890	ape
8080	llama-server	8085	privacy-shield
8090	embeddings	8188	comfyui
8585	brave-search	8880	tts
8888	searxng	9000	whisper
9120	hermes-proxy	host network	tailscale

Extension System

Every bundled service manifest lives under extensions/services/<id>/ with:

• manifest.yaml — Service contract (id, port, health endpoint, category, GPU backends, dependencies, features)
• compose.yaml — Docker Compose service definition (optional; core services live in docker-compose.base.yml)
• compose.{nvidia,amd}.yaml — GPU-specific overlays
• Dockerfile — Custom image build (if needed)

The manifest schema is enforced by extensions/schema/service-manifest.v1.json. The service registry library (lib/service-registry.sh) provides lookup functions for the CLI and installer.

CI/CD

Workflow	Purpose
test-linux.yml	Integration suite: smoke, manifests, health, BATS, tier map, contracts
matrix-smoke.yml	Multi-distro smoke (Ubuntu, Debian, Fedora, Arch, openSUSE)
validate-compose.yml	Docker Compose file validation
validate-env.yml	Environment variable schema validation
dashboard.yml	Dashboard build and lint
lint-shell.yml	ShellCheck on all .sh files
lint-python.yml	Python linting (ruff, black)
type-check-python.yml	Python type checking (mypy)
secret-scan.yml	GitLeaks secret detection
lint-powershell.yml	PowerShell linting for Windows installer

Design Principles

Priority when principles conflict: Let It Crash > KISS > Pure Functions > SOLID

• Let It Crash: No broad catches, no silent swallowing. Errors propagate visibly. Bash uses set -euo pipefail everywhere.
• KISS: Readable over clever. One function, one job. No premature abstraction.
• Pure Functions: Installer libraries (installers/lib/) are the pure functional core; phases are the imperative shell.
• SOLID: Extend via config/data (manifests, backend JSON), not code modification.