opensandbox-group--opensandbox/docs/guides/multi-tenancy.md

title, description

title	description
Multi-Tenancy	Isolate sandbox workloads across Kubernetes namespaces with per-tenant API keys and automatic namespace routing.

Multi-Tenancy

Multi-tenancy enables a single OpenSandbox Server deployment to serve multiple independent consumers, each isolated in its own Kubernetes namespace. Each tenant gets dedicated API key(s), and all sandbox lifecycle operations (create, list, get, delete) are automatically scoped to the tenant's namespace.

Requirements

• opensandbox-server >= 0.2.2
• Kubernetes runtime (runtime.type = "kubernetes" in server.toml)
• One Kubernetes namespace per tenant, pre-created by a cluster admin
• Default Helm chart RBAC (already includes ClusterRole with cross-namespace access)

::: tip Proxy routes in multi-tenant mode Unlike single-tenant mode, proxy routes require OPEN-SANDBOX-API-KEY when multi-tenancy is enabled. Ensure clients include the header for proxy requests. :::

::: warning Docker not supported Multi-tenancy requires Kubernetes namespace isolation. If runtime.type = "docker" is configured and tenants are enabled, the server refuses to start with a fatal error. :::

How It Works

Request with OPEN-SANDBOX-API-KEY header
       │
       ├── Tenant provider configured?
       │       ├── YES → lookup key in tenant provider
       │       │       ├── found  → inject tenant context, route to tenant namespace
       │       │       └── not found → 401 Unauthorized
       │       └── NO  → validate against server.api_key (single-tenant legacy)

When multi-tenancy is active:

1. Auth middleware resolves the API key to a TenantEntry via the configured provider.
2. The tenant's namespace is injected into a request-scoped ContextVar.
3. Sandbox lifecycle operations (create/list/get/delete) use the resolved namespace.
4. List operations only return sandboxes within the authenticated tenant's namespace.
5. Proxy routes (/sandboxes/{id}/proxy/...) also require OPEN-SANDBOX-API-KEY in multi-tenant mode.

Configuration

Multi-tenancy is enabled by adding a [tenants] section to server.toml. Two provider backends are available: file (local TOML) and http (remote endpoint).

File Provider (default)

Add to server.toml:

[tenants]
provider = "file"

Then create tenants.toml at one of these paths (checked in order):

1. $SANDBOX_TENANTS_CONFIG_PATH environment variable
2. ~/.opensandbox/tenants.toml

Example tenants.toml:

[[tenants]]
name = "team-alpha"
namespace = "sandbox-alpha"
api_keys = ["sk-alpha-key-1", "sk-alpha-key-2"]

[[tenants]]
name = "team-beta"
namespace = "sandbox-beta"
api_keys = ["sk-beta-key-1"]

Each tenant entry requires:

Field	Description
name	Unique human-readable tenant identifier
namespace	Target Kubernetes namespace (must exist at server startup)
api_keys	One or more API keys for this tenant (supports key rotation)

Constraints:

• API keys must be unique across all tenants (duplicate → startup error).
• Every tenant must have at least one key.
• server.api_key in server.toml must be removed when using [tenants].

HTTP Provider

For environments where tenants are managed by an external IAM or tenant management system:

[tenants]
provider = "http"
endpoint = "https://iam.internal/opensandbox/tenant-lookup"
max_stale_seconds = 300.0
timeout_seconds = 5.0
auth_header = "X-Internal-Auth"
auth_token = "bearer-token-here"

Field	Default	Description
provider	"file"	"file" or "http"
endpoint	—	Required for HTTP. URL the server GETs to resolve a key.
max_stale_seconds	300.0	Max seconds to serve stale cache when endpoint is unreachable.
timeout_seconds	5.0	HTTP request timeout.
auth_header	—	Optional header name for provider-level auth.
auth_token	—	Optional token value for that header.

HTTP endpoint contract:

Request:

GET {endpoint}
Header: OPEN-SANDBOX-API-KEY: <api_key>

Response (200):

{
  "namespace": "sandbox-alpha",
  "ttl": 60
}

Response (401):

{
  "code": "UNAUTHORIZED",
  "message": "Invalid API key"
}

The HTTP provider caches results per key using the server-suggested ttl. On TTL expiry it re-fetches synchronously. If the endpoint is unreachable, stale entries are served up to max_stale_seconds, after which requests fail with 503.

Namespace Setup

Before onboarding a tenant, the cluster admin must prepare the target namespace.

1. Create namespace

kubectl create namespace sandbox-alpha

2. ResourceQuota (recommended)

apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
  namespace: sandbox-alpha
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi
    pods: "20"

3. LimitRange (recommended)

apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-limits
  namespace: sandbox-alpha
spec:
  limits:
  - default:
      cpu: "1"
      memory: 2Gi
    defaultRequest:
      cpu: 250m
      memory: 512Mi
    type: Container

5. Server RBAC

The default Helm chart (opensandbox-server) already deploys a ClusterRole + ClusterRoleBinding with cross-namespace access. No additional RBAC changes are needed for multi-tenancy — the server ServiceAccount can already operate in any namespace.

Hot Reload

File provider

The file provider watches tenants.toml for changes (polls every 2 seconds). Changes take effect without server restart:

• New key added → immediately valid on next request.
• Key removed → immediately invalid (401).
• Parse error during reload → previous config retained, warning logged.
• File deleted → all tenant keys invalidated.

The watcher monitors the parent directory to handle Kubernetes ConfigMap atomic symlink swaps.

HTTP provider

No explicit reload needed — cache entries expire per their ttl and are re-fetched on next lookup.

Key Rotation

Multi-tenancy supports multiple API keys per tenant specifically for zero-downtime key rotation:

1. Add the new key to tenants.toml (or issue it via HTTP provider).
2. Wait for hot-reload (file) or TTL expiry (HTTP).
3. Update clients to use the new key.
4. Remove the old key from config.

Both old and new keys are valid simultaneously during the transition window.

Migration from Single-Tenant

1. Create target namespace(s) for your tenant(s).
2. Write tenants.toml with your existing API key as a tenant entry, pointing to the existing namespace:

   [[tenants]]
   name = "default"
   namespace = "your-existing-namespace"
   api_keys = ["your-existing-api-key"]
   

3. Add [tenants] section to server.toml:

   [tenants]
   provider = "file"
   

4. Remove api_key from [server] in server.toml.
5. Deploy. The existing key continues working, now as a tenant key.
6. Add more tenants as needed.

Rollback: Remove the [tenants] section from server.toml, restore server.api_key, restart.

Isolation Model

The server itself does not enforce resource quotas or network policies. Isolation is delegated to Kubernetes:

Dimension	Kubernetes Mechanism	Scope
Resource quota	ResourceQuota	Per-namespace CPU, memory, storage
Default limits	LimitRange	Per-namespace default container resources
Network isolation	Egress sidecar	Per-sandbox outbound policy via egress proxy
Sandbox count	ResourceQuota (pod count)	Per-namespace pod limit
RBAC	RoleBinding	Per-namespace API access

::: info Pool APIs are not tenant-scoped Pool management routes (/pools) currently operate in the server's configured default namespace, not the authenticated tenant's namespace. Pools are shared resources. If you need per-tenant pool isolation, create separate server deployments or wait for future per-tenant pool support. :::

Troubleshooting

Symptom	Cause	Fix
Server exits at startup: "Multi-tenancy requires Kubernetes namespaces"	runtime.type = "docker" with [tenants] configured	Use Kubernetes runtime or remove [tenants]
Server exits: "Remove server.api_key from server.toml"	Both server.api_key and [tenants] present	Remove api_key from [server] section
Server exits: namespace not found	Tenant namespace doesn't exist	kubectl create namespace <ns> before starting server
401 on valid key after config change	Hot-reload hasn't picked up change yet	Wait 2s (file) or TTL seconds (HTTP)
503 TENANT_PROVIDER_UNAVAILABLE	HTTP endpoint unreachable and cache expired past max_stale_seconds	Fix endpoint connectivity; increase max_stale_seconds
Duplicate api_key error at startup	Same key assigned to multiple tenants	Ensure each key is unique across all tenants