opensandbox-group--opensandbox/docs/guides/secure-container.md

---
title: Secure Container Runtime
description: Use gVisor, Kata Containers, and Firecracker microVMs for hardware-level sandbox isolation.
---

# Secure Container Runtime Guide

This guide explains how to use secure container runtimes with OpenSandbox to provide hardware-level isolation for executing untrusted AI-generated code.

## Table of Contents

- [Overview](#overview)
- [Server Configuration](#server-configuration)
- [Docker Mode](#docker-mode)
- [Kubernetes Mode](#kubernetes-mode)
- [User Guide](#user-guide)
- [Administrator Guide](#administrator-guide)
- [Troubleshooting and Best Practices](#troubleshooting-and-best-practices)

---

## Overview

### What are Secure Container Runtimes?

Secure container runtimes provide stronger isolation than the standard runc runtime used by Docker and containerd. They add additional security layers through different mechanisms:

| Runtime | Isolation Mechanism | Startup Overhead | Memory Overhead | Best For |
|---------|---------------------|------------------|-----------------|----------|
| **runc** (default) | Process-level cgroups | ~0ms | Minimal | Trusted workloads, local development |
| **gVisor** | User-space kernel (syscall interception) | ~10-50ms | ~50MB | General workloads with low overhead |
| **Kata (QEMU)** | Full VM with QEMU hypervisor | ~500ms | ~20-50MB | Maximum compatibility and isolation |
| **Kata (Firecracker)** | MicroVM with Firecracker hypervisor | ~125ms | ~5MB | High density, minimal footprint |
| **Kata (CLH)** | Cloud Hypervisor | ~200ms | ~10-20MB | Balanced performance and isolation |

### Why Use Secure Runtimes?

OpenSandbox is designed to execute untrusted code generated by AI models (Claude, GPT-4, Gemini, etc.). Secure runtimes provide:

1. **Container Escape Protection**: Prevents malicious code from breaking out of the container
2. **Kernel-Level Isolation**: Each sandbox gets its own kernel context
3. **Multi-Tenant Safety**: Different users' sandboxes are strongly isolated
4. **Compliance**: Meets security requirements for regulated industries

### Supported Runtime Types

OpenSandbox supports the following secure runtime types through server-level configuration:

- `"gvisor"` - Google gVisor with runsc
- `"kata"` - Kata Containers with QEMU hypervisor (default)
- `"firecracker"` - Kata Containers with Firecracker hypervisor
- `""` (empty) - Standard runc (default, no secure runtime)

### Key Design Principle

**Server-Level Configuration**: The secure runtime is configured once at the server level by administrators. All sandboxes on that server transparently use the configured runtime. SDK users and API callers require **no code changes**.

---

## Server Configuration

Secure runtimes are configured through the `~/.sandbox.toml` configuration file. The server validates the configured runtime at startup and will refuse to start if the runtime is unavailable.

### Configuration File

Edit `~/.sandbox.toml`:

```toml
[runtime]
type = "docker"  # or "kubernetes"
execd_image = "opensandbox/execd:latest"

# Secure container runtime configuration
# When enabled, ALL sandboxes on this server use the specified runtime
[secure_runtime]
# Runtime type: "", "gvisor", "kata", "firecracker"
type = ""

# Docker mode: OCI runtime name (e.g., "runsc" for gVisor, "kata-runtime" for Kata)
# Required when runtime.type = "docker" and type is not empty
docker_runtime = "runsc"

# Kubernetes mode: RuntimeClass name (e.g., "gvisor", "kata-qemu", "kata-fc")
# Required when runtime.type = "kubernetes" and type is not empty
k8s_runtime_class = "gvisor"
```

### Configuration Examples

#### Example 1: gVisor on Docker

```toml
[runtime]
type = "docker"
execd_image = "opensandbox/execd:latest"

[secure_runtime]
type = "gvisor"
docker_runtime = "runsc"
k8s_runtime_class = "gvisor"
```

#### Example 2: Kata Containers on Kubernetes

```toml
[runtime]
type = "kubernetes"
execd_image = "opensandbox/execd:latest"

[secure_runtime]
type = "kata"
docker_runtime = "kata-runtime"
k8s_runtime_class = "kata-qemu"
```

#### Example 3: Kata + Firecracker on Kubernetes

```toml
[runtime]
type = "kubernetes"
execd_image = "opensandbox/execd:latest"

[secure_runtime]
type = "firecracker"
docker_runtime = ""  # Not supported in Docker mode
k8s_runtime_class = "kata-fc"
```

### Startup Validation

When the server starts, it automatically validates that the configured secure runtime is available:

```bash
$ opensandbox-server
INFO     Validating secure runtime for Docker backend
INFO     Docker OCI runtime 'runsc' is available: {...}
INFO     Application startup complete.
```

If the runtime is not available, the server will refuse to start with a clear error message:

```
ERROR    Configured Docker runtime 'runsc' is not available.
        Available runtimes: runc.
        Please install and configure it in /etc/docker/daemon.json.
```

---

## Docker Mode

Docker mode is fully supported for secure container runtimes.

### Prerequisites

- Docker daemon installed and running
- Secure runtime installed on the host

### gVisor Setup for Docker

#### Step 1: Install gVisor runsc

For Docker mode, you only need to install the **runsc** OCI runtime:

```bash
# Ubuntu/Debian
curl -fsSL https://gvisor.dev/archive.key | sudo gpg --dearmor -o /usr/share/keyrings/gvisor-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/gvisor-archive-keyring.gpg] https://storage.googleapis.com/gvisor/releases release main" | \
  sudo tee /etc/apt/sources.list.d/gvisor.list
sudo apt-get update && sudo apt-get install -y runsc

# Verify installation
runsc --version
```

> **Note**: For Docker mode, only `runsc` is required. The `containerd-shim-runsc-v1` is only needed for Kubernetes/containerd.
>
> **Reference**: See [gVisor Installation Guide](https://gvisor.dev/docs/user_guide/install/) for other distributions and installation methods.

#### Step 2: Configure Docker daemon

Use the `runsc install` command to automatically configure Docker daemon:

```bash
sudo runsc install
```

Or manually edit `/etc/docker/daemon.json`:

```json
{
  "runtimes": {
    "runsc": {
      "path": "/usr/bin/runsc",
      "runtimeArgs": [
        "--platform=systrap",
        "--network=host"
      ]
    }
  }
}
```

Restart Docker:

```bash
sudo systemctl restart docker
```

> **Reference**: See [gVisor Docker Quick Start](https://gvisor.dev/docs/user_guide/quick_start/docker/) for more details.

#### Step 3: Configure OpenSandbox Server

Edit `~/.sandbox.toml`:

```toml
[runtime]
type = "docker"
execd_image = "opensandbox/execd:latest"

[secure_runtime]
type = "gvisor"
docker_runtime = "runsc"
```

#### Step 4: Start Server and Verify

```bash
opensandbox-server
```

Create a test sandbox:

```bash
curl -X POST http://localhost:8080/v1/sandboxes \
  -H "Content-Type: application/json" \
  -d '{
    "image": {"uri": "python:3.11"},
    "timeout": 3600,
    "resourceLimits": {"cpu": "500m", "memory": "512Mi"},
    "entrypoint": ["python", "-u", "-c", "import time\nwhile True: print('hello from gVisor!'); time.sleep(1)"],
    "metadata": {
      "name": "gvisor-docker-sandbox"
    }
  }'
```

Verify the runtime:

```bash
docker ps --format "{{.ID}}\t{{.Image}}\t{{.Names}}"
docker inspect <container_id> | grep -A2 Runtime
# Expected output:
# "Runtime": "runsc",
```

### Kata Containers Setup for Docker

#### System Requirements

Kata Containers requires hardware virtualization support. Verify your system meets the following requirements:

**Hardware Virtualization Support:**
```bash
# Check if CPU supports hardware virtualization (VT-x for Intel, AMD-V for AMD)
lscpu | grep Virtualization
# Expected output: Virtualization: VT-x (Intel) or AMD-V (AMD)

# Alternatively on Intel
grep -E --color=auto 'vmx|svm' /proc/cpuinfo
# Expected: vmx (Intel) or svm (AMD) flags present
```

**KVM Module:**
```bash
# Check if KVM module is loaded
lsmod | grep kvm
# Expected: kvm_intel (Intel) or kvm_amd (AMD)

# If not loaded, load KVM module
sudo modprobe kvm_intel  # For Intel
# or
sudo modprobe kvm_amd    # For AMD
```

**Kernel Requirements:**
- Linux kernel 5.10 or later recommended
- KVM enabled in kernel config

**Docker Requirements:**
- Docker 20.10 or later
- `/etc/docker/daemon.json` configured for Kata runtime

#### Installation

Download and install Kata Containers static binaries from GitHub releases:

```bash
# Find the latest release at https://github.com/kata-containers/kata-containers/releases
KATA_VERSION="3.27.0"
wget https://github.com/kata-containers/kata-containers/releases/download/${KATA_VERSION}/kata-static-${KATA_VERSION}-amd64.tar.zst

# Extract to root directory - Kata will be installed in /opt/kata
zstd -d kata-static-${KATA_VERSION}-amd64.tar.zst
tar -xvf kata-static-${KATA_VERSION}-amd64.tar -C /

# Create symbolic links for PATH access
sudo ln -sf /opt/kata/bin/kata-runtime /usr/local/bin/kata-runtime
sudo ln -sf /opt/kata/bin/containerd-shim-kata-v2 /usr/local/bin/containerd-shim-kata-v2

# Verify installation
kata-runtime --version
```

#### Configure Docker Daemon

Edit `/etc/docker/daemon.json` to register Kata as a runtime:

```json
{
  "default-runtime": "runc",
  "runtimes": {
    "kata": {
      "runtimeType": "io.containerd.kata.v2"
    }
  }
}
```

Restart Docker to apply changes:

```bash
sudo systemctl restart docker

# Verify Kata is available in Docker
docker info | grep -A5 Runtimes
# Expected output should include "io.containerd.runc.v2 kata"
```

#### Configure OpenSandbox Server

Edit `~/.sandbox.toml`:

```toml
[runtime]
type = "docker"
execd_image = "opensandbox/execd:latest"

[secure_runtime]
type = "kata"
docker_runtime = "kata"
```

#### Verify Installation

**Test with OpenSandbox API**

Create a sandbox and verify it's running in a VM by checking the kernel:

```bash
# Create a test sandbox
curl --location 'http://127.0.0.1:8080/v1/sandboxes' \
  --header 'Content-Type: application/json' \
  --data '{
    "image": {"uri": "ubuntu:latest"},
    "timeout": 3600,
    "resourceLimits": {"cpu": "500m", "memory": "512Mi"},
    "entrypoint": ["/bin/bash", "-c", "while true; do uname -a; sleep 1; done"],
    "metadata": {
      "name": "kata-sandbox"
    }
  }'
```

Check the container's kernel to verify VM isolation:

```bash
# Get the container ID
docker ps | grep kata-sandbox

# Check the kernel inside the container (should be different from host)
docker exec <container_id> uname -a
# Expected output: Linux <hostname> 5.10.x-generic #x86_64 ... (Kata VM kernel)

# Compare with host kernel
uname -a
# Host kernel might be different version or have different hostname
```

**Key Indicators of Kata VM:**
- Container runs in a separate kernel with different hostname
- Kernel version is typically `5.10.x` (Kata's guest kernel)
- Host process list shows `qemu-system-x86_64` or similar hypervisor process
---

## Kubernetes Mode

Kubernetes mode supports secure runtimes through RuntimeClass resources.

### Prerequisites

- Kubernetes cluster with containerd runtime
- Secure runtime installed on all nodes
- RuntimeClass CRDs created

### gVisor Setup for Kubernetes

#### Step 1: Install gVisor Components on All Nodes

For Kubernetes with containerd, you need to install **two** components:

1. **runsc** - the gVisor OCI runtime
2. **containerd-shim-runsc-v1** - the containerd shim for gVisor

```bash
# On each node - Ubuntu/Debian
curl -fsSL https://gvisor.dev/archive.key | sudo gpg --dearmor -o /usr/share/keyrings/gvisor-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/gvisor-archive-keyring.gpg] https://storage.googleapis.com/gvisor/releases release main" | \
  sudo tee /etc/apt/sources.list.d/gvisor.list
sudo apt-get update

# Install both gVisor components
sudo apt-get install -y runsc containerd-shim-runsc-v1

# Verify installation
runsc --version
containerd-shim-runsc-v1 --version
```
> **Reference**: See [gVisor Installation Guide](https://gvisor.dev/docs/user_guide/containerd/configuration/) for complete installation instructions and other distributions.

#### Step 2: Configure containerd

Edit `/etc/containerd/config.toml`:

```toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc]
          runtime_type = "io.containerd.runsc.v1"
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc.options]
            TypeUrl = "io.containerd.runsc.v1.options"
            ConfigPath = "/etc/containerd/runsc.toml"
```

```bash
sudo tee /etc/containerd/runsc.toml > /dev/null <<'EOF'
[runsc]
  platform = "ptrace"
EOF
```

Restart containerd:

```bash
sudo systemctl restart containerd
```

#### Step 3: Create RuntimeClass CRD

```yaml
# gvisor-runtimeclass.yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
scheduling:
  nodeSelector:
    kubernetes.io/arch: amd64
```

```bash
kubectl apply -f gvisor-runtimeclass.yaml
```

#### Step 4: Configure OpenSandbox Server

Edit `~/.sandbox.toml`:

```toml
[runtime]
type = "kubernetes"
execd_image = "opensandbox/execd:latest"

[secure_runtime]
type = "gvisor"
k8s_runtime_class = "gvisor"
```

#### Step 5: Verify Installation

```bash
# Test the RuntimeClass
kubectl run test-gvisor --restart=Never --image=hello-world --runtime-class=gvisor
kubectl logs test-gvisor
kubectl delete pod test-gvisor
```

### Kata Containers Setup for Kubernetes

#### Step 1: Install Kata Containers

Follow the [official Kata Containers installation guide](https://github.com/kata-containers/kata-containers/blob/main/tools/packaging/kata-deploy/helm-chart/README.md).

Quick installation using Helm:

```bash
# Install kata-deploy which will set up Kata Containers via DaemonSet
helm install kata-deploy "oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy" --version "3.27.0" --namespace kube-system --create-namespace

# Wait for kata-deploy pods to be ready
kubectl wait --for=condition=ready pod -l name=kata-deploy -n kube-system --timeout=300s
```

> **Note**: The `kata-deploy` DaemonSet will automatically configure containerd on all nodes. Manual containerd configuration is not required when using kata-deploy.

#### Step 2: Verify Installation

Check that Kata Containers is installed and RuntimeClasses are created:

```bash
# Check RuntimeClasses
kubectl get runtimeclass

# Expected output:
# NAME         HANDLER     AGE
# kata         kata-qemu   10m
# kata-qemu    kata-qemu   10m
# kata-clh     kata-clh    10m
# kata-fc      kata-fc     10m

# Test Kata with a simple pod
kubectl run test-kata --restart=Never --image=hello-world --runtime-class=kata-qemu
kubectl logs test-kata
kubectl delete pod test-kata
```

### Creating Pools for Different Runtimes (Optional)

When using Pool CRDs for pre-warmed sandboxes, create separate pools for each runtime type:

```yaml
# gvisor-pool.yaml
apiVersion: sandbox.opensandbox.io/v1alpha1
kind: Pool
metadata:
  name: gvisor-pool
  labels:
    runtime: gvisor
spec:
  template:
    spec:
      runtimeClassName: gvisor
      containers:
        - name: sandbox-container
          image: opensandbox/code-interpreter:v1.1.0
  capacitySpec:
    bufferMax: 10
    bufferMin: 2
    poolMax: 20
    poolMin: 5
```

---

## User Guide

This section is for AI application developers using OpenSandbox.

### No Code Changes Required

**Important**: The secure runtime is configured at the server level. Your code does not need to change.

Simply create a sandbox using the OpenSandbox Lifecycle API - the server automatically applies the configured secure runtime:

**Create a test sandbox:**

```bash
curl -X POST http://localhost:8080/v1/sandboxes \
  -H "Content-Type: application/json" \
  -d '{
    "image": {"uri": "python:3.11"},
    "timeout": 3600,
    "resourceLimits": {"cpu": "500m", "memory": "512Mi"},
    "entrypoint": ["python", "-u", "-c", "import time\nwhile True: print(\"hello from secure sandbox!\"); time.sleep(1)"],
    "metadata": {
      "name": "my-secure-sandbox"
    }
  }'
```

**Response:**

```json
{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "status": "running"
}
```

The sandbox will automatically use the secure runtime configured on the server (gVisor, Kata, or runc).

### How It Works

1. **Administrator** configures the secure runtime in `~/.sandbox.toml`
2. **Server** validates the runtime at startup
3. **Server** automatically injects the runtime into each sandbox:
   - Docker mode: Adds `runtime` to HostConfig
   - Kubernetes mode: Adds `runtimeClassName` to Pod spec
4. **User** creates sandboxes via API - no runtime parameter needed

### Verifying Runtime Isolation

After creating a sandbox, verify the runtime being used:

**Docker mode:**
```bash
docker ps --format "{{.ID}}\t{{.Image}}\t{{.Names}}"
docker inspect <container_id> | grep -A2 Runtime
# Expected output for gVisor:
# "Runtime": "runsc",
```

**Kubernetes mode:**
```bash
kubectl get pod <pod-name> -o jsonpath='{.spec.runtimeClassName}'
# Expected output for gVisor:
# gvisor
```

---

## Administrator Guide

This section is for platform operators and SREs managing secure runtime infrastructure.

### Prerequisites

Secure runtimes must be installed and configured on your infrastructure **before** configuring OpenSandbox. OpenSandbox does not install runtimes automatically.

### Installation Summary

| Runtime | Docker | Kubernetes |
|---------|--------|------------|
| gVisor | Install runsc → Configure daemon.json | Install runsc → Configure containerd → Create RuntimeClass |
| Kata (QEMU) | Install kata-runtime → Configure daemon.json | Install Kata → Configure containerd → Create RuntimeClass |
| Kata (Firecracker) | Not supported | Install Kata → Configure containerd → Create RuntimeClass |

### Configuration Validation

The server validates secure runtime configuration at startup:

1. **Docker mode**: Checks if the runtime exists in Docker daemon's runtime list
2. **Kubernetes mode**: Checks if the RuntimeClass exists in the cluster

If validation fails, the server refuses to start with a clear error message.

### Security Best Practices

1. **Default to gVisor**: Provides good security with acceptable performance for most workloads
2. **Use Kata for Untrusted Code**: Maximum isolation for completely unknown code
3. **Regular Updates**: Keep runtimes updated for security patches
4. **Test Compatibility**: Validate your workloads with the chosen runtime before production
5. **Monitor Resources**: Secure runtimes have higher memory overhead

### Runtime Selection Guidelines

| Use Case | Recommended Runtime | Reasoning |
|----------|---------------------|-----------|
| Development/Testing | runc (default) | Fastest startup, lowest overhead |
| Production AI Code Execution | gVisor | Good balance of security and performance |
| High-Security Requirements | Kata (QEMU) | Maximum isolation, full compatibility |
| High-Density Multi-Tenant | Kata (Firecracker) | Minimal memory overhead per sandbox |
| Untrusted Network Code | gVisor or Kata | Syscall filtering prevents network attacks |

---

## Troubleshooting and Best Practices

### Common Issues

#### 1. Runtime Not Found (Docker)

**Error**: `Configured Docker runtime 'runsc' is not available.`

**Solution**: Ensure the runtime is configured in `/etc/docker/daemon.json` and Docker has been restarted:

```bash
sudo systemctl restart docker
docker info | grep -A5 Runtimes
```

#### 2. RuntimeClass Not Found (Kubernetes)

**Error**: `RuntimeClass 'gvisor' does not exist.`

**Solution**: Create the RuntimeClass CRD:

```bash
kubectl get runtimeclass
kubectl apply -f gvisor-runtimeclass.yaml
```

#### 3. Syscall Compatibility Issues

**Error**: Container exits with code 1, no logs

**Cause**: gVisor doesn't implement all syscalls. Some applications may not be compatible.

**Solution**: Check the [gVisor compatibility guide](https://gvisor.dev/docs/user_guide/compatibility/). Try using Kata (QEMU) which has better compatibility.

#### 4. Pod Stuck in ContainerCreating

**Cause**: RuntimeClass handler not configured on the node.

**Solution**: Verify containerd configuration:

```bash
# On the node
sudo containerd config dump
sudo systemctl restart containerd
```

#### 5. Egress Sidecar Incompatible with gVisor

**Error**: Sandbox pods CrashLoopBackOff with egress container log:
```
iptables: Failed to initialize nft: Protocol not supported
```
Or with iptables-legacy:
```
iptables v1.8.9 (legacy): can't initialize iptables table 'nat': Table does not exist (do you need to insmod?)
```

**Cause**: gVisor's netstack implements the `filter` and `mangle` iptables tables but does not implement the `nat` table. The egress sidecar uses a REDIRECT rule in the `nat` table to intercept DNS queries (port 53 → 15353), so it cannot start under gVisor. This is an upstream gVisor limitation ([gvisor#170](https://github.com/google/gvisor/issues/170)).

**Solution**:
- Use `secure_runtime.type = "kata"` with `k8s_runtime_class = "kata-qemu"` — Kata provides a full Linux kernel per pod, so the `nat` table is available and the egress sidecar works unchanged.
- Use a CNI-level FQDN policy (e.g., Cilium `toFQDNs`) instead of the egress sidecar for network isolation under gVisor.
- Remove `network_policy` from sandbox creation requests if egress control is not required.

> **Note**: The server validates this combination at request time and returns HTTP 400 with a clear error message when `secure_runtime.type = "gvisor"` and `network_policy` are used together.

### Compatibility Matrix

| Feature | runc | gVisor | Kata (QEMU) | Kata (CLH) | Kata (FC) |
|---------|------|--------|-------------|------------|-----------|
| Syscall Compatibility | Full | Partial | Full | Full | Limited |
| GPU Support | Yes | No | Yes | Yes | No |
| IPv6 | Yes | Yes | Yes | Yes | Yes |
| Privileged Mode | Yes | No | Yes | Yes | No |
| Docker Volume | Yes | Yes | Yes | Yes | Yes |
| Systemd | Yes | No | Yes | Yes | No |
| iptables `nat` table (egress sidecar) | Yes | **No** | Yes | Yes | Yes |

### Getting Help

- **Documentation**: [OpenSandbox GitHub](https://github.com/opensandbox-group/OpenSandbox)
- **Issues**: Report bugs via [GitHub Issues](https://github.com/opensandbox-group/OpenSandbox/issues)
- **Design Document**: See [OSEP-0004](https://github.com/opensandbox-group/OpenSandbox/blob/main/oseps/0004-secure-container-runtime.md) for complete design details