opensandbox-group--opensandbox/docs/components/egress.md

title, description

title	description
Egress	FQDN-based egress control sidecar for OpenSandbox providing DNS filtering, nftables enforcement, and credential injection.

OpenSandbox Egress Sidecar

The Egress is a core component of OpenSandbox that provides FQDN-based egress control.

It runs alongside the sandbox application container (sharing the same network namespace) and enforces declared network policies.

Features

• FQDN-based Allowlist: Control outbound traffic by domain name (e.g., api.github.com).
• IP / CIDR Targets: Egress rules can also target literal IP addresses or CIDR ranges (e.g., 10.0.0.0/8).
• Wildcard Support: Allow subdomains using wildcards (e.g., *.pypi.org).
• Transparent Interception: Uses transparent DNS proxying; no application configuration required.
• Experimental: Transparent HTTPS MITM (mitmproxy): Optional transparent TLS interception for outbound 80/443 traffic in the sidecar network namespace.
• Dynamic DNS (dns+nft mode): When a domain is allowed and the proxy resolves it, the resolved A/AAAA IPs are added to nftables with TTL so that default-deny + domain-allow is enforced at the network layer.
• Credential Vault: Automatic credential injection (bearer, basic, API-key, custom headers, and scoped placeholder substitutions) for allowed hosts via transparent mitmproxy. See Credential Vault.
• Privilege Isolation: Requires CAP_NET_ADMIN only for the sidecar; the application container runs unprivileged.
• Fail-Closed Enforcement: DNS redirect setup is required through iptables or the native nft fallback; the sidecar exits if no enforced redirect can be installed. Optional subsystems (OpenTelemetry, startup hooks) degrade gracefully.

Architecture

The egress control is implemented as a Sidecar that shares the network namespace with the sandbox application.

1. DNS Proxy (Layer 1):

   • Runs on 127.0.0.1:15353.
   • iptables rules redirect all port 53 (DNS) traffic to this proxy.
   • Filters queries based on the allowlist.
   • Returns NXDOMAIN for denied domains.

2. Network Filter (Layer 2) (when OPENSANDBOX_EGRESS_MODE=dns+nft):

   • Uses nftables to enforce IP-level allow/deny. Resolved IPs for allowed domains are added to dynamic allow sets with TTL (dynamic DNS).
   • At startup, the sidecar whitelists 127.0.0.1 (redirect target for the proxy) and nameserver IPs from /etc/resolv.conf so DNS resolution and proxy upstream work (including private DNS). Nameserver count is capped and invalid IPs are filtered.

Kubernetes Service Access Under defaultAction: deny

In Kubernetes deployments that use defaultAction: deny, reaching an in-cluster Service usually needs two separate allowances:

• allow the Service DNS name so the DNS proxy resolves it
• allow the Service CIDR (or a narrower ClusterIP range) so dns+nft does not drop the TCP connection after resolution

Allowing only postgres.opensandbox.svc.cluster.local is not sufficient if the resolved ClusterIP still belongs to a denied range such as 10.96.0.0/12. Likewise, allowing only the CIDR is not sufficient if the DNS proxy still denies the hostname.

See Network Isolation for operator guidance and examples.

Requirements

• Runtime: Docker or Kubernetes.
• Capabilities: CAP_NET_ADMIN (for the sidecar container only).
• Kernel: Linux kernel with iptables support.
• Service mesh: OpenSandbox egress is not currently supported inside pods that already have a transparent service-mesh sidecar (for example Istio/Envoy injection). Both layers rewrite outbound traffic in the same network namespace and can conflict.

Configuration

Most deployments only need these settings:

• Mode: OPENSANDBOX_EGRESS_MODE
  • dns (default): DNS filtering only
  • dns+nft: DNS + nftables IP/CIDR enforcement (recommended for strict default-deny)
• Initial policy:
  • OPENSANDBOX_EGRESS_RULES (JSON, same shape as POST /policy)
  • or OPENSANDBOX_EGRESS_POLICY_FILE (if valid file exists, it takes precedence at startup)
• HTTP API:
  • OPENSANDBOX_EGRESS_HTTP_ADDR (default :18080)
  • OPENSANDBOX_EGRESS_TOKEN (optional auth via OPENSANDBOX-EGRESS-AUTH)
• Rule limit:
  • OPENSANDBOX_EGRESS_MAX_RULES for POST/PATCH /policy (default 4096, 0 disables cap)

Optional advanced features:

• Nameserver bypass: OPENSANDBOX_EGRESS_NAMESERVER_EXEMPT
• Denied hostname webhook: OPENSANDBOX_EGRESS_DENY_WEBHOOK, OPENSANDBOX_EGRESS_SANDBOX_ID
• DoH/DoT controls: OPENSANDBOX_EGRESS_BLOCK_DOH_443, OPENSANDBOX_EGRESS_DOH_BLOCKLIST
• Custom DNS upstream: OPENSANDBOX_EGRESS_DNS_UPSTREAM (comma-separated IPs, optional :port), OPENSANDBOX_EGRESS_DNS_UPSTREAM_TIMEOUT (default 5 seconds)
• DNS upstream health probe: OPENSANDBOX_EGRESS_DNS_UPSTREAM_PROBE (enable), OPENSANDBOX_EGRESS_DNS_UPSTREAM_PROBE_INTERVAL_SEC
• Credential vault: OPENSANDBOX_EGRESS_CREDENTIAL_VAULT_REQUIRE_TLS, OPENSANDBOX_EGRESS_CREDENTIAL_VAULT_TRUSTED_PROXY_CIDRS, OPENSANDBOX_CREDENTIAL_PROXY_SOCKET (default /run/opensandbox/credential-proxy/active.sock)
• Metrics: OPENSANDBOX_EGRESS_METRICS_EXTRA_ATTRS (extra key=value attributes for OTLP metrics and structured log fields)

Always-Rules Files

Static rule files under /var/egress/rules/ are loaded at startup and take priority over dynamic API rules:

File	Purpose
/var/egress/rules/deny.always	Domains always denied, overrides user and allow rules
/var/egress/rules/allow.always	Domains always allowed, overrides user rules
/var/egress/rules/log_skip.always	Domain patterns whose DNS blocks are not logged (noise reduction)

Format: one domain per line (supports wildcards like *.example.com). Lines starting with # are comments. Missing files are silently ignored.

Rule precedence: deny.always > allow.always > user policy (API/env).

Always-rules are hot-reloaded: the sidecar polls the files once per minute and applies changes without restart.

Service Mesh Compatibility

::: warning Not Supported with Transparent Mesh Sidecars OpenSandbox egress is designed to be the only transparent outbound interception layer inside the sandbox pod. Deployments that automatically inject a service-mesh sidecar such as Istio/Envoy into the same pod are not currently supported for egress-sidecar features. :::

Why this conflicts today:

• OpenSandbox egress installs iptables/nft redirect rules in the shared pod network namespace so DNS and optional HTTPS MITM traffic flow through the egress sidecar.
• Service meshes such as Istio also redirect outbound traffic in that same namespace, usually to Envoy.
• When both are present, the redirect order becomes deployment-dependent and can produce double interception, broken TLS, or traffic that bypasses the expected Credential Vault / egress-policy path.

This matters for:

• per-sandbox networkPolicy / network_policy enforcement
• transparent mitmproxy mode
• Credential Vault / Credential Proxy

Recommended operator choices today:

1. Exclude OpenSandbox sandbox pods from automatic mesh sidecar injection when they need the egress sidecar.
2. If mesh injection is mandatory, do not rely on the OpenSandbox egress sidecar for outbound control in those pods; instead use a platform-level mechanism such as a CNI/network-policy solution.
3. Treat mesh-injected sandboxes as a separate runtime profile and document that Credential Vault and transparent egress interception are unavailable there until first-class coexistence support is implemented.

See also Credential Vault and Network Isolation.

Runtime HTTP API

Method	Path	Description
GET	/policy	Get current policy and enforcement mode
POST	/policy	Replace policy ({}, null, empty body => reset to deny-all)
PUT	/policy	Alias for POST
PATCH	/policy	Merge/append rules (body is JSON array of egress rules)
DELETE	/policy	Remove specific targets (body is JSON string array, e.g. ["*.example.com"])
GET/POST/PATCH/DELETE	/credential-vault	Manage the credential vault (create, update, delete)
GET	/credential-vault/credentials	List credential metadata
GET	/credential-vault/credentials/{name}	Get single credential metadata
GET	/credential-vault/bindings	List binding metadata
GET	/credential-vault/bindings/{name}	Get single binding metadata
GET	/healthz	Health check; returns 200 ok or 503 mitmproxy not ready (when transparent MITM is enabled but not yet initialized)

Quick example:

# Replace policy
curl -XPOST http://127.0.0.1:18080/policy \
  -d '{"defaultAction":"deny","egress":[{"action":"allow","target":"*.example.com"}]}'

# Remove specific targets
curl -XDELETE http://127.0.0.1:18080/policy \
  -d '["*.example.com"]'

Experimental: Transparent MITM (mitmproxy)

::: warning Experimental APIs, environment variables, and behavior may change. :::

Optional transparent HTTPS interception for outbound 80/443 traffic in the sidecar network namespace.

Credential Vault

The credential vault provides automatic credential injection for outbound requests to allowed hosts. Credentials are stored in-memory and injected into matching requests by the transparent mitmproxy layer.

Prerequisites: transparent mitmproxy enabled (OPENSANDBOX_EGRESS_MITMPROXY_TRANSPARENT=true), egress API auth token set (OPENSANDBOX_EGRESS_TOKEN).

Supported auth types: bearer, basic, apiKey, customHeaders.

See Credential Vault for full API usage, binding rules, and security model.

Observability (OpenTelemetry)

Egress can export OTLP metrics; application logs use the native zap logger (JSON to stdout by default, configurable via OPENSANDBOX_LOG_OUTPUT / OPENSANDBOX_EGRESS_LOG_LEVEL). OTLP log export is not used.

Build & Run

Build Docker Image

cd components/egress

# Build locally
docker build -t opensandbox/egress:local .

# Or use the build script (multi-arch)
./build.sh

Run Locally

1. Start sidecar:

docker run -d --name sandbox-egress \
  --cap-add=NET_ADMIN \
  opensandbox/egress:local

2. Apply policy:

curl -XPOST http://127.0.0.1:18080/policy \
  -d '{"defaultAction":"deny","egress":[{"action":"allow","target":"*.google.com"}]}'

3. Run app container in the same network namespace:

docker run --rm -it \
  --network container:sandbox-egress \
  curlimages/curl sh

4. Verify from app container:

curl -I https://google.com
curl -I https://github.com

Development

• Language: Go 1.25+
• Key Packages:
  • pkg/dnsproxy: DNS server and policy matching logic.
  • pkg/iptables: iptables rule management.
  • pkg/nftables: nftables static/dynamic rules and DNS-resolved IP sets.
  • pkg/policy: Policy parsing and definition.
  • pkg/credentialvault: Credential vault store and binding validation.
  • pkg/startup: Post-startup hook registry (Register/RunPost).
  • hooks/: Side-effect import target; init() functions register startup hooks that run after iptables/MITM setup.

cd components/egress
go test ./...

Process Supervisor

The egress container runs under opensandbox-supervisor, a lightweight process wrapper that restarts the egress worker on crash with exponential backoff, a crashloop circuit breaker, and structured JSONL event logging.

ENTRYPOINT: supervisor --pre-start=cleanup.sh --name=egress --grace-period=20s -- /opt/opensandbox-egress/egress

Egress-specific configuration:

• --grace-period=20s: Egress needs extra time to drain DNS connections and tear down iptables/nft rules on shutdown (default is 10 s).
• Pre-start hook (cleanup.sh): Reaps orphaned mitmdump processes from a previous crash and removes stale DNS redirect iptables/native nft state that would otherwise point port 53 at a dead proxy. It does not manage the inet opensandbox policy table; the nftables manager deletes and recreates that table when policy enforcement starts.

Troubleshooting

• "iptables setup failed": ensure sidecar has --cap-add=NET_ADMIN.
• DNS fails for all domains: check sidecar upstream DNS reachability and logs.
• Traffic not blocked as expected: in dns+nft, verify nft applied (nft list table inet opensandbox) and check sidecar logs for fallback.