# Omnigent on Kubernetes Deploy Omnigent to any Kubernetes cluster using Kustomize. The manifests pull the prebuilt image and set up a persistent volume and health checks. They also include an Ingress so you can serve the app over HTTPS at a public web address, but that part is optional — it only matters when people need to reach the server over the internet, and it pulls in two extra add-ons (ingress-nginx and cert-manager). For local or dev use, ignore it and connect with `kubectl port-forward` (see [Verify the deployment](#verify-the-deployment)). ## What gets provisioned - **Deployment** — single-replica pod running `ghcr.io/omnigent-ai/omnigent-server`, served on port 8000. - **Service** — ClusterIP on port 80 → 8000. - **Ingress** *(optional)* — serves the app over HTTPS at a public web address, using cert-manager for the certificate. Skip it if the server isn't going on the internet. - **PVC** — 10 Gi volume at `/data/artifacts` for the artifact store, minted cookie secret, and admin credentials. - **ConfigMap + Secret** — environment config and database credentials. ## Prerequisites - A Kubernetes cluster (1.25+) - `kubectl` with Kustomize support (`kubectl kustomize` or standalone `kustomize`) - A PostgreSQL database (managed or in-cluster — see below) - *Only if you're putting the server on a public web address:* an ingress controller (e.g. ingress-nginx) and cert-manager ### Install cluster add-ons for ingress and cert management (optional) Skip this unless you're putting the server on a public web address. (For local or dev use you'll reach it with `kubectl port-forward`, or you can let your own load balancer or proxy handle HTTPS instead.) Otherwise, if your cluster doesn't already have an ingress controller and cert-manager, install them (pin the versions to taste): ```bash # ingress-nginx — use the provider manifest that matches your cluster # (this is the kind one; for EKS/GKE/AKS use that provider's manifest or Helm chart): kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml # cert-manager: kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml # wait until both are ready: kubectl wait -n ingress-nginx --for=condition=Ready pod \ -l app.kubernetes.io/component=controller --timeout=180s kubectl wait -n cert-manager --for=condition=Available deployment --all --timeout=180s ``` ### Create a cert-manager issuer (optional) Skip this unless you're using the Ingress. cert-manager fetches the HTTPS certificate for the Ingress from a `ClusterIssuer` named `letsencrypt-prod` (the `cert-manager.io/cluster-issuer` annotation in `base/ingress.yaml`). That issuer is **not** shipped here — create one before deploying, or change the annotation to match an issuer you already have. Two common choices: ```yaml # Production — real certificates from Let's Encrypt # (needs a public domain and an Ingress reachable from the internet): apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: you@example.com privateKeySecretRef: name: letsencrypt-prod solvers: - http01: ingress: ingressClassName: nginx ``` ```yaml # Local / dev — self-signed (no public DNS needed; browsers will warn): apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: selfSigned: {} ``` Apply your chosen issuer with `kubectl apply -f `. Without it, cert-manager logs `IssuerNotFound` and no certificate is issued (the server still runs — only TLS is affected). ## Deploy with an external database Use this path when you have a managed Postgres (RDS, Cloud SQL, Neon, etc.). 1. **Edit the secret** — set your real `DATABASE_URL` and generate a cookie secret: ```bash # deploy/kubernetes/base/secret.yaml DATABASE_URL: "postgresql+psycopg://user:pass@your-db-host:5432/omnigent" OMNIGENT_ACCOUNTS_COOKIE_SECRET: "$(openssl rand -hex 32)" ``` 2. **Set your domain** *(skip if you're not using the Ingress)* — replace `omnigent.example.com` in `base/ingress.yaml` with your domain, and make sure the `letsencrypt-prod` ClusterIssuer exists (see [Create a cert-manager issuer](#create-a-cert-manager-issuer)). 3. **Apply:** ```bash kubectl kustomize deploy/kubernetes/base/ | kubectl apply -f - ``` 4. **Create the first admin.** Open the app (via your Ingress host, or port-forward for a quick check — see [Verify the deployment](#verify-the-deployment)). With the default `accounts` provider the first visitor claims the instance: the Setup screen prompts for a username + password, and whoever finishes it first becomes the admin. ## Deploy with in-cluster Postgres The `overlays/postgres/` overlay adds a single-replica Postgres 16 StatefulSet with its own 10 Gi PVC. Good for dev/testing clusters. 1. **Edit secrets** — in `overlays/postgres/secret-patch.yaml`, replace `changeme` with real passwords: ```bash POSTGRES_PASSWORD: "" DATABASE_URL: "postgresql+psycopg://omnigent:@postgres:5432/omnigent" OMNIGENT_ACCOUNTS_COOKIE_SECRET: "$(openssl rand -hex 32)" ``` 2. **Set your domain** *(skip if you're not using the Ingress)* — edit the hostname in `base/ingress.yaml`, and make sure the `letsencrypt-prod` ClusterIssuer exists (see [Create a cert-manager issuer](#create-a-cert-manager-issuer)). 3. **Apply:** ```bash kubectl kustomize deploy/kubernetes/overlays/postgres/ | kubectl apply -f - ``` ## Deploy with OpenShell sandboxes The `overlays/openshell/` overlay configures the server to provision [NVIDIA OpenShell](https://github.com/NVIDIA/OpenShell) sandboxes for managed sessions, and includes RBAC for the [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) CRD when the gateway uses a Kubernetes compute driver. 1. **Edit the configmap patch** — set `OMNIGENT_SANDBOX_SERVER_URL` to the public URL sandboxes will dial back to, and optionally set `OPENSHELL_GATEWAY` to a specific gateway name: ```bash # deploy/kubernetes/overlays/openshell/configmap-patch.yaml OMNIGENT_SANDBOX_SERVER_URL: "https://omnigent.example.com" OPENSHELL_GATEWAY: "my-gateway" ``` 2. **Edit secrets** — in `overlays/openshell/secret-patch.yaml`, set the database URL, cookie secret, and the LLM API keys your harness needs: ```bash DATABASE_URL: "postgresql+psycopg://omnigent:@your-db-host:5432/omnigent" OMNIGENT_ACCOUNTS_COOKIE_SECRET: "$(openssl rand -hex 32)" ANTHROPIC_API_KEY: "sk-ant-..." ``` 3. **Gateway access** — the server pod needs to reach the OpenShell gateway's gRPC endpoint. If the gateway runs in-cluster, make sure the NetworkPolicy allows it (the included policy allows all egress on 443 — tighten to taste). If the gateway stores its config/TLS material in a Secret, create `openshell-gateway-config` in the `omnigent` namespace and the deployment mounts it at `~/.config/openshell`. 4. **Install the agent-sandbox CRD** *(optional)* — if the OpenShell gateway delegates to the kubernetes-sigs/agent-sandbox controller: ```bash kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/agent-sandbox/main/config/crd/bases/sandbox.agent.k8s.io_agentsandboxes.yaml ``` The overlay's RBAC already grants the server's ServiceAccount permission to manage `AgentSandbox` resources. 5. **Apply:** ```bash kubectl kustomize deploy/kubernetes/overlays/openshell/ | kubectl apply -f - ``` For OpenShell + in-cluster Postgres, layer the postgres overlay on top (compose both bases in a new kustomization, or apply the postgres StatefulSet separately). See [Network egress policy](../openshell/README.md#network-egress-policy) for the sandbox-side egress allow-list (server URL + LLM provider hosts). ## Building a UBI image (Red Hat / OpenShift) For RHEL and OpenShift environments that require UBI-compliant containers, use the UBI variant of the Dockerfile. It uses Red Hat Universal Base Image 9 (`ubi9/python-312`, `ubi9/nodejs-20`) and runs the server as non-root (UID 1001) by default — compatible with OpenShift's `restricted-v2` SCC out of the box. ```bash # from the repo root docker build -t omnigent-server:ubi -f deploy/docker/Dockerfile.ubi . ``` Then reference the image in the OpenShift overlay by patching the Deployment or pointing your image stream at it. ## Deploy on Red Hat OpenShift The `overlays/openshift/` overlay replaces the Ingress with an OpenShift Route (edge TLS, managed by the platform) and adds a `restricted-v2`-compatible SecurityContext. No ingress controller or cert-manager add-ons needed. 1. **Edit the secret** in `base/secret.yaml` (same as the external-database path above). 2. **Set your route hostname** — replace `omnigent.apps.example.com` in `overlays/openshift/route.yaml` with your cluster's apps domain. 3. **Apply:** ```bash kubectl kustomize deploy/kubernetes/overlays/openshift/ | oc apply -f - ``` For in-cluster Postgres on OpenShift, use `overlays/openshift-postgres/` instead — it combines the Postgres StatefulSet, OpenShift Route, and restricted security contexts: ```bash # edit overlays/openshift-postgres/secret-patch.yaml with real passwords first kubectl kustomize deploy/kubernetes/overlays/openshift-postgres/ | oc apply -f - ``` ## On-demand sandbox runners The `overlays/sandbox-runners/` overlay turns on the **`kubernetes`** managed sandbox provider: a `host_type: managed` session spawns one runner Pod that runs `omnigent host` as its entrypoint and dials back over the launch-token tunnel. It adds a dedicated runner namespace, a least-privilege server SA (scoped Pod + Secret rights, **no `pods/exec`**), and the `sandbox:` server config. The overlay swaps in the official `omnigent-server-kubernetes` image variant, which adds the `kubernetes` client extra the provider imports (the base server image omits it). See `overlays/sandbox-runners/README.md` for the full guide. ```bash kubectl apply -k deploy/kubernetes/overlays/sandbox-runners # then create the omnigent-creds harness Secret (see the overlay README) ``` **Credentials & auth** — two separate concerns, don't conflate: - **Server auth.** Front the server with `header`/`oidc` auth or run single-user; the built-in `accounts` mode refuses the per-session runner dial-back (`403`), a framework-level limit shared by all sandbox providers — see [Auth](../README.md#auth). - **Model keys** (`ANTHROPIC_API_KEY` / `CLAUDE_CODE_OAUTH_TOKEN` / `OPENAI_API_KEY` / `GIT_TOKEN` / …) ride the `omnigent-creds` Secret projected into every runner Pod. Both are detailed in [`overlays/sandbox-runners/README.md`](overlays/sandbox-runners/README.md#server-auth-managed-hosts). ## Verify the deployment Check the rollout and reach the server without a public domain: ```bash kubectl get pods -n omnigent # omnigent (and, with the overlay, postgres) → Running kubectl rollout status deploy/omnigent -n omnigent kubectl logs -n omnigent deploy/omnigent # server logs # Port-forward the Service and open the app locally: kubectl port-forward -n omnigent svc/omnigent 8000:80 # → http://localhost:8000 (health check: curl localhost:8000/health → {"status":"ok"}) ``` The first boot runs database migrations before the server starts listening; the pod may restart once if the liveness probe fires during that window (see [Resource sizing](#resource-sizing)). To test the Ingress itself instead of port-forwarding, point its hostname at a domain that already resolves to localhost — `omnigent.localtest.me` or `.sslip.io` — use the self-signed issuer above, and reach it through the ingress controller's published port. ## Next steps: connect a host The server is the control plane — agents run on **hosts** that register with it. A brand-new deployment has none, so connect at least one machine: ```bash omnigent login https://omnigent.example.com # authenticate the CLI omnigent host --server https://omnigent.example.com # register this machine ``` The host then appears in the web UI when you start a new chat. See the [main README](../../README.md) for the full host/auth reference. ## Use your own IdP instead (OIDC) — optional Optional. The default `accounts` provider (username + password) works out of the box; use this only to delegate authentication to an external OIDC provider. Add OIDC env vars to the secret: ```bash kubectl create secret generic omnigent-oidc -n omnigent \ --from-literal=OMNIGENT_AUTH_PROVIDER=oidc \ --from-literal=OMNIGENT_OIDC_ISSUER=https://github.com \ --from-literal=OMNIGENT_OIDC_CLIENT_ID= \ --from-literal=OMNIGENT_OIDC_CLIENT_SECRET= \ --from-literal=OMNIGENT_OIDC_REDIRECT_URI=https://omnigent.example.com/auth/callback \ --from-literal=OMNIGENT_OIDC_COOKIE_SECRET=$(openssl rand -hex 32) ``` Then add `envFrom: [{secretRef: {name: omnigent-oidc}}]` to the Deployment container spec (or merge the values into `omnigent-secrets`). ## Resource sizing The server idles around ~275 MB RSS. The manifests request 512 Mi and limit at 1 Gi — adjust to taste. The first boot against a remote Postgres runs migrations and takes ~1 minute; bump the liveness `initialDelaySeconds` to ~90s if you see the pod get killed during the first deploy. ## Scaling The server uses an in-memory runner registry, so **only one replica is supported**. Do not increase `replicas` unless the architecture is changed to use a shared registry (e.g. Redis).