chore: import upstream snapshot with attribution

helm/sim/.claude/skills/sim-helm/references/install-paths.md

@@ -0,0 +1,192 @@
+# Install Path Selection
+
+Three mutually-exclusive paths for the app Secret. Pick exactly one. The chart enforces this at template time.
+
+## Decision tree
+
+```
+Is this a production install?
+├── No (dev / kind / minikube / dry-run)
+│     → Inline `--set` is fine. Skip to "Path A".
+│
+└── Yes
+      │
+      Do you already manage secrets with Vault / AWS Secrets Manager /
+      Azure Key Vault / GCP Secret Manager / 1Password Connect?
+      │
+      ├── Yes → External Secrets Operator. Path C.
+      │
+      └── No
+            │
+            Do you use GitOps with Sealed Secrets, SOPS, or
+            hand-managed Kubernetes Secrets?
+            │
+            ├── Yes → Pre-existing Secret. Path B.
+            │
+            └── No → Install ESO and go to Path C.
+                    (Don't skip to inline `--set` for prod —
+                    secrets land in `helm get values` and release history.)
+```
+
+---
+
+## Path A — Inline `--set` (dev only)
+
+```bash
+helm install sim ./helm/sim \
+  --namespace sim --create-namespace \
+  --set app.env.BETTER_AUTH_SECRET=$(openssl rand -hex 32) \
+  --set app.env.ENCRYPTION_KEY=$(openssl rand -hex 32) \
+  --set app.env.INTERNAL_API_SECRET=$(openssl rand -hex 32) \
+  --set app.env.CRON_SECRET=$(openssl rand -hex 32) \
+  --set postgresql.auth.password=$(openssl rand -base64 24 | tr -d '/+=')
+```
+
+The chart generates a `Secret` named `<release>-app-secrets` containing every non-empty key from `app.env` + `realtime.env`. Both `app` and `realtime` Deployments mount it via `envFrom`.
+
+**Risks:**
+- Secrets are visible in `helm get values <release>` and `helm history <release>`.
+- Anyone with read access to the release's ConfigMap (`sh.helm.release.v1.<release>.v<N>`) can recover the secrets — they're stored base64-encoded inside.
+
+---
+
+## Path B — Pre-existing Kubernetes Secret
+
+Create the Secret first, then point the chart at it.
+
+```bash
+kubectl create namespace sim
+kubectl create secret generic sim-app-secrets --namespace sim \
+  --from-literal=BETTER_AUTH_SECRET=$(openssl rand -hex 32) \
+  --from-literal=ENCRYPTION_KEY=$(openssl rand -hex 32) \
+  --from-literal=INTERNAL_API_SECRET=$(openssl rand -hex 32) \
+  --from-literal=CRON_SECRET=$(openssl rand -hex 32)
+
+kubectl create secret generic sim-postgres-secret --namespace sim \
+  --from-literal=POSTGRES_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=')
+```
+
+```yaml
+# values.yaml
+app:
+  secrets:
+    existingSecret:
+      enabled: true
+      name: sim-app-secrets
+
+postgresql:
+  auth:
+    existingSecret:
+      enabled: true
+      name: sim-postgres-secret
+      passwordKey: POSTGRES_PASSWORD
+```
+
+**The chart cannot introspect your Secret.** If you forget a required key, the pod will fail at runtime with `CreateContainerConfigError: secret key "X" not found`. The required keys are: `BETTER_AUTH_SECRET`, `ENCRYPTION_KEY`, `INTERNAL_API_SECRET`, plus `CRON_SECRET` when cronjobs are enabled.
+
+For GitOps (Sealed Secrets / SOPS), seal/encrypt the Secret YAML before committing — never commit a plain `kubectl create secret` output.
+
+---
+
+## Path C — External Secrets Operator (production recommended)
+
+ESO syncs from your existing secret store (Vault, AWS SM, Azure KV, GCP SM, etc.) into a Kubernetes Secret on a refresh interval. The chart renders the `ExternalSecret` resource; ESO does the syncing.
+
+### Prerequisites
+
+1. Install ESO once per cluster:
+   ```bash
+   helm repo add external-secrets https://charts.external-secrets.io
+   helm install external-secrets external-secrets/external-secrets \
+     -n external-secrets --create-namespace
+   ```
+2. Create a `ClusterSecretStore` (or namespace-scoped `SecretStore`) that points at your secret manager. ESO's docs cover the auth wiring for each provider.
+
+### Values
+
+```yaml
+externalSecrets:
+  enabled: true
+  apiVersion: v1beta1     # v1beta1 works on ESO >= 0.7. Bump to v1 only on ESO >= 0.17.
+  refreshInterval: 1h
+  secretStoreRef:
+    name: my-cluster-secret-store
+    kind: ClusterSecretStore     # or SecretStore for namespace-scoped
+  remoteRefs:
+    app:
+      BETTER_AUTH_SECRET: sim/app/better-auth-secret
+      ENCRYPTION_KEY: sim/app/encryption-key
+      INTERNAL_API_SECRET: sim/app/internal-api-secret
+      CRON_SECRET: sim/app/cron-secret     # required iff cronjobs.enabled
+      # Optional but commonly mapped:
+      API_ENCRYPTION_KEY: sim/app/api-encryption-key
+      OPENAI_API_KEY: sim/providers/openai
+    postgresql:
+      password: sim/postgresql/password    # required if postgresql.enabled
+    externalDatabase:
+      password: sim/postgresql/password    # required if externalDatabase.enabled
+
+# Leave app.env empty (or only set non-secret values like NEXT_PUBLIC_APP_URL).
+app:
+  env: {}
+```
+
+### Fail-fast behavior
+
+The chart will refuse to render if:
+
+- `externalSecrets.enabled=true` and any of `BETTER_AUTH_SECRET`, `ENCRYPTION_KEY`, `INTERNAL_API_SECRET` (or `CRON_SECRET` when cronjobs are enabled) is **neither** set in `app.env` **nor** mapped in `remoteRefs.app`. Error message names the missing key.
+- A key is set in `app.env` with a non-empty value but not mapped in `remoteRefs.app` (would be silently dropped from the rendered Secret).
+
+These checks catch the "renders cleanly, CrashLoopBackOffs at runtime" failure mode that plagued earlier chart versions.
+
+### Remote ref shapes
+
+Each `remoteRefs.app.<KEY>` value can be either:
+
+```yaml
+# Shorthand — just the path/key in the store
+BETTER_AUTH_SECRET: sim/app/better-auth-secret
+```
+
+```yaml
+# Full form — pass any field ESO supports
+BETTER_AUTH_SECRET:
+  key: sim/app/better-auth-secret
+  property: value          # for stores that return JSON
+  version: "v3"            # pin a specific version
+  decodingStrategy: Base64 # for base64-stored values
+```
+
+---
+
+## Cross-cutting: things that are NOT secrets
+
+Operational tunables (rate limits, timeouts, IVM pool size, branding) live in `app.envDefaults` and `realtime.envDefaults`. They're rendered as **inline `env:`** on the Deployment, not written to the Secret. See `values-model.md` for the full mental model.
+
+Don't try to push these into ESO — they're not sensitive, they'd just bloat the secret store.
+
+---
+
+## Verifying your choice
+
+After `helm install`:
+
+```bash
+# What Secret will the pods mount?
+helm template sim helm/sim -f my-values.yaml | grep -A2 "envFrom:"
+
+# For ESO: did the ExternalSecret render?
+helm template sim helm/sim -f my-values.yaml | grep -B1 -A10 "kind: ExternalSecret"
+
+# For existingSecret: is your pre-created Secret referenced?
+helm template sim helm/sim -f my-values.yaml | grep -E "name: .*-app-secrets"
+```
+
+For ESO, after `helm install`, verify the sync:
+
+```bash
+kubectl get externalsecret -n sim
+kubectl describe externalsecret <release>-app-secrets -n sim
+# Status should show 'SecretSynced=True'
+```

helm/sim/.claude/skills/sim-helm/references/secrets.md

@@ -0,0 +1,88 @@
+# Secret Generation
+
+The Sim chart requires four cryptographic secrets at install time. Generate them once and store them in your chosen path (see `install-paths.md`). Never reuse these across environments.
+
+## Generate all four at once
+
+```bash
+export BETTER_AUTH_SECRET=$(openssl rand -hex 32)
+export ENCRYPTION_KEY=$(openssl rand -hex 32)
+export INTERNAL_API_SECRET=$(openssl rand -hex 32)
+export CRON_SECRET=$(openssl rand -hex 32)
+
+# Optional but commonly needed:
+export API_ENCRYPTION_KEY=$(openssl rand -hex 32)       # MUST be exactly 64 hex chars
+export POSTGRES_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=')  # if using chart-bundled Postgres
+```
+
+## What each secret does
+
+| Key | Purpose | Length | Rotation impact |
+|---|---|---|---|
+| `BETTER_AUTH_SECRET` | Signs user session JWTs (Better Auth) | 32 bytes = 64 hex chars | Rotating invalidates all active sessions — users must re-login |
+| `ENCRYPTION_KEY` | App-level encryption for sensitive fields | 32 bytes = 64 hex chars | Rotating breaks decryption of existing data — requires migration |
+| `INTERNAL_API_SECRET` | Shared auth between `sim-app` ↔ `sim-realtime` pods | 32 bytes = 64 hex chars | Both deployments must roll together — temporary realtime errors during the rollout |
+| `CRON_SECRET` | Authenticates scheduled CronJob pods to the app | 32 bytes = 64 hex chars | Rotating just needs `helm upgrade`; next cron run uses the new value |
+| `API_ENCRYPTION_KEY` (optional) | Encrypts user-stored API keys (OpenAI tokens, etc.) at rest in Postgres | **Exactly 64 hex chars** (the app rejects other lengths) | Without it, keys are stored plain. Once set, never rotate without a migration |
+| `POSTGRES_PASSWORD` (chart-bundled Postgres only) | Postgres superuser password | Any length ≥ 12 chars matching `^[a-zA-Z0-9._-]+$` | Requires Postgres pod restart + app rollout |
+
+The `^[a-zA-Z0-9._-]+$` constraint on the Postgres password exists because the chart embeds the password into `DATABASE_URL` without URL-encoding. The `tr -d '/+='` strips the three problematic characters from `openssl rand -base64` output. The chart enforces this regex at template time.
+
+## Storage by path
+
+### Path A (inline `--set`)
+
+Pass each on the command line — see `install-paths.md` Path A.
+
+### Path B (pre-existing Kubernetes Secret)
+
+```bash
+kubectl create namespace sim
+kubectl create secret generic sim-app-secrets --namespace sim \
+  --from-literal=BETTER_AUTH_SECRET=$BETTER_AUTH_SECRET \
+  --from-literal=ENCRYPTION_KEY=$ENCRYPTION_KEY \
+  --from-literal=INTERNAL_API_SECRET=$INTERNAL_API_SECRET \
+  --from-literal=CRON_SECRET=$CRON_SECRET \
+  --from-literal=API_ENCRYPTION_KEY=$API_ENCRYPTION_KEY
+
+kubectl create secret generic sim-postgres-secret --namespace sim \
+  --from-literal=POSTGRES_PASSWORD=$POSTGRES_PASSWORD
+```
+
+For GitOps, run the `kubectl create secret ... --dry-run=client -o yaml` and pipe through `kubeseal` (Sealed Secrets) or `sops` before committing.
+
+### Path C (External Secrets Operator)
+
+Push the generated values into your secret manager first. Example for AWS Secrets Manager:
+
+```bash
+aws secretsmanager create-secret --name sim/app/better-auth-secret --secret-string "$BETTER_AUTH_SECRET"
+aws secretsmanager create-secret --name sim/app/encryption-key     --secret-string "$ENCRYPTION_KEY"
+aws secretsmanager create-secret --name sim/app/internal-api-secret --secret-string "$INTERNAL_API_SECRET"
+aws secretsmanager create-secret --name sim/app/cron-secret        --secret-string "$CRON_SECRET"
+aws secretsmanager create-secret --name sim/app/api-encryption-key --secret-string "$API_ENCRYPTION_KEY"
+aws secretsmanager create-secret --name sim/postgresql/password    --secret-string "$POSTGRES_PASSWORD"
+```
+
+Then map the paths in `externalSecrets.remoteRefs.app` (see `install-paths.md` Path C).
+
+## Rotation
+
+Sim doesn't have built-in rotation hooks. The procedure is:
+
+1. Generate a new value, store it.
+2. `helm upgrade` (or let ESO pick up the change on its next refresh).
+3. Restart the affected workloads to force re-read of `envFrom`:
+   ```bash
+   kubectl rollout restart deploy/sim-app deploy/sim-realtime -n sim
+   ```
+4. For `BETTER_AUTH_SECRET`: expect a wave of `401`s as old sessions invalidate.
+5. For `ENCRYPTION_KEY` / `API_ENCRYPTION_KEY`: **do not rotate** without an explicit data migration. Existing ciphertext becomes undecryptable.
+
+## What NOT to do
+
+- **Don't reuse the same secret across dev/staging/prod.** A leak in one tier compromises all.
+- **Don't commit secrets to git, even in private repos.** Use sealed-secrets / SOPS / ESO.
+- **Don't paste secrets into Slack, Discord, GitHub issues, or screenshots.** Treat them like database passwords.
+- **Don't store secrets in `values.yaml` files committed to git.** That's worse than `--set` — values files persist forever in history.
+- **Don't generate secrets with weak entropy.** No `date | md5`, no `password123`, no developer's birthday. `openssl rand` or `/dev/urandom` only.

helm/sim/.claude/skills/sim-helm/references/troubleshooting.md

@@ -0,0 +1,183 @@
+# Troubleshooting
+
+Map symptom → root cause → fix. Always run the diagnostic block first, then match.
+
+## Diagnostic block (run this first)
+
+```bash
+NS=sim  # adjust to your namespace
+kubectl --namespace $NS get pods,events --sort-by='.lastTimestamp'
+kubectl --namespace $NS describe pod -l app.kubernetes.io/instance=sim
+kubectl --namespace $NS logs deploy/sim-app --tail=200
+kubectl --namespace $NS logs deploy/sim-realtime --tail=200
+kubectl --namespace $NS logs sts/sim-postgresql --tail=200
+kubectl --namespace $NS logs job/sim-migrations --tail=200 2>/dev/null || true
+```
+
+---
+
+## `Error: execution error at (sim/templates/...): app.env.BETTER_AUTH_SECRET is required for production deployment`
+
+**Cause:** `helm install` / `helm upgrade` ran without the required secrets set.
+
+**Fix:** Generate and pass the four required secrets (see `references/secrets.md`). For production, switch to `existingSecret` or ESO instead of `--set` (see `references/install-paths.md`).
+
+---
+
+## `Error: execution error ...: Required key 'X' is missing: externalSecrets.enabled=true but the key is neither set in app.env nor mapped in externalSecrets.remoteRefs.app`
+
+**Cause:** ESO is enabled but one of the required keys (`BETTER_AUTH_SECRET`, `ENCRYPTION_KEY`, `INTERNAL_API_SECRET`, or `CRON_SECRET` when cronjobs are enabled) isn't mapped via `remoteRefs.app`. The chart fails fast at template time to avoid CrashLoopBackOff later.
+
+**Fix:** Add the mapping:
+
+```yaml
+externalSecrets:
+  remoteRefs:
+    app:
+      <KEY>: path/in/your/secret/store
+```
+
+Or, if you really don't need cronjobs, set `cronjobs.enabled=false` to drop the `CRON_SECRET` requirement.
+
+---
+
+## `Error: execution error ...: Key 'X' is set in app.env but externalSecrets.enabled=true and externalSecrets.remoteRefs.app.X is not configured`
+
+**Cause:** ESO is enabled. The chart-managed Secret is not rendered. A key set in `app.env` would be silently dropped — pods would start with the wrong (missing) value.
+
+**Fix:** Either map the key via `remoteRefs.app.X` so ESO syncs it, OR remove the key from `app.env` if you don't need it.
+
+---
+
+## App pods stuck in `CrashLoopBackOff`
+
+Get the logs first:
+
+```bash
+kubectl logs --namespace sim deploy/sim-app --tail 200
+```
+
+Match the error:
+
+| Log line | Cause | Fix |
+|---|---|---|
+| `Invalid env: ... NEXT_PUBLIC_APP_URL: Invalid url` | URL field set to empty string or invalid format | Set `app.env.NEXT_PUBLIC_APP_URL` to a valid URL — `https://sim.example.com` in prod, `http://localhost:3000` in dev |
+| `getaddrinfo ENOTFOUND ... -postgresql` / `connect ECONNREFUSED` | App can't reach Postgres | Check `kubectl get pod -l app.kubernetes.io/name=postgresql` is `Running`; check `postgresql.auth.password` matches the password in the Secret |
+| `password authentication failed for user "sim"` | Postgres password rotated but app pod wasn't restarted, OR password contains URL-unsafe chars | `kubectl rollout restart deploy/sim-app -n sim`; regenerate password with `openssl rand -base64 24 \| tr -d '/+='` |
+| `BETTER_AUTH_SECRET is missing` / `INTERNAL_API_SECRET is required` | Required env var not present in the Secret | Verify with `kubectl get secret sim-app-secrets -o jsonpath='{.data}' \| jq 'keys'`; if missing, fix your secret strategy |
+| `Migration failed` or app starts before migration | Migration Job hasn't completed | `kubectl logs job/sim-migrations -n sim`; rerun with `kubectl delete job/sim-migrations && helm upgrade ...` |
+
+---
+
+## Image pull errors (`ErrImagePull` / `ImagePullBackOff`)
+
+```bash
+kubectl describe pod -l app.kubernetes.io/name=sim -n sim | grep -A5 "Failed\|Warning"
+```
+
+| Cause | Fix |
+|---|---|
+| Private registry, no pull secret | Set `global.imagePullSecrets: [{name: my-regcred}]` and create the regcred Secret: `kubectl create secret docker-registry my-regcred --docker-server=... --docker-username=... --docker-password=...` |
+| Image tag doesn't exist in the registry | `helm get values sim`, check the rendered `image.tag`; correct it or fall back to `Chart.AppVersion` |
+| Air-gapped cluster | Mirror the image to your internal registry, set `global.imageRegistry=my-registry.example.com` |
+
+---
+
+## Postgres pod `Pending`
+
+```bash
+kubectl describe pvc --namespace sim
+```
+
+Always one of:
+
+| `Events` says | Cause | Fix |
+|---|---|---|
+| `no persistent volumes available for this claim and no storage class is set` | No default StorageClass | Set `global.storageClass: <your-class>` or annotate one as default |
+| `Failed to provision volume with StorageClass "X"` | No PV provisioner installed | Install one (`local-path-provisioner` for kind, EBS CSI for EKS, PD CSI for GKE, Azure Disk CSI for AKS) |
+| `only ReadWriteOnce access modes are supported` | StorageClass doesn't support RWO | Pick a different `global.storageClass` |
+| `pod has unbound immediate PersistentVolumeClaims` and no events | StorageClass uses `WaitForFirstConsumer` and pod isn't schedulable | Check pod's `nodeSelector` / `tolerations` against your nodes |
+
+---
+
+## Ingress not routing
+
+```bash
+kubectl get ingress -n sim
+kubectl describe ingress -n sim
+```
+
+| Cause | Fix |
+|---|---|
+| No `ADDRESS` in `kubectl get ingress` | Ingress controller not installed — install `ingress-nginx`, AWS LBC, GCP LB controller, etc. |
+| `ingressClassName` doesn't match installed controller | `kubectl get ingressclass` to list installed classes, set `ingress.className` to match |
+| Address is set but DNS resolves to wrong IP | `dig <your-host>` — point DNS at the ingress controller's external IP / LoadBalancer / CNAME |
+| TLS cert errors | If using cert-manager, check `kubectl describe certificate -n sim`; verify `ingress.tls.issuerRef` |
+| `503 Service Unavailable` | Ingress routing is fine but app pod isn't `Ready` — go back to the diagnostic block |
+
+---
+
+## CronJob pods fail with `CreateContainerConfigError: couldn't find key CRON_SECRET in Secret`
+
+**Cause:** `cronjobs.enabled=true` (the default) but `CRON_SECRET` isn't in the app Secret. Two paths:
+
+1. Inline mode: `app.env.CRON_SECRET=""` — the chart will fail at template time. If you somehow got past that, regenerate and set it.
+2. Existing-Secret mode: your pre-created Secret doesn't include `CRON_SECRET`. Add it:
+   ```bash
+   kubectl patch secret sim-app-secrets -n sim --type='json' \
+     -p='[{"op":"add","path":"/data/CRON_SECRET","value":"'$(openssl rand -hex 32 | base64)'"}]'
+   ```
+3. ESO mode: missing `remoteRefs.app.CRON_SECRET` mapping. Add it.
+
+Or set `cronjobs.enabled=false` if you don't need scheduled jobs.
+
+---
+
+## `app.kubernetes.io/managed-by: Helm` collisions when upgrading
+
+You installed once with `--name foo`, then tried to install again with `--name bar` into the same namespace. Resources collide on labels.
+
+**Fix:** Use distinct namespaces per release, or `helm uninstall foo -n <ns>` first.
+
+---
+
+## Pods get OOMKilled
+
+```bash
+kubectl get events -n sim --field-selector reason=OOMKilling
+```
+
+Bump the relevant resource limit. Defaults:
+
+| Workload | Default request | Default limit |
+|---|---|---|
+| `app` | `1000m` CPU / `4Gi` memory | `2000m` CPU / `8Gi` memory |
+| `realtime` | `250m` CPU / `512Mi` memory | `500m` CPU / `1Gi` memory |
+| `postgresql` | `250m` CPU / `512Mi` memory | `1000m` CPU / `2Gi` memory |
+
+Override in values:
+
+```yaml
+app:
+  resources:
+    requests:
+      memory: 8Gi
+    limits:
+      memory: 16Gi
+```
+
+---
+
+## Logs to grab when filing a support issue
+
+```bash
+helm version
+kubectl version --short
+helm get values sim -n sim --revision $(helm history sim -n sim | tail -1 | awk '{print $1}')
+kubectl get all,pvc,ingress,externalsecret -n sim -o wide
+kubectl describe pods -n sim -l app.kubernetes.io/instance=sim | head -200
+kubectl logs --tail=500 -n sim deploy/sim-app
+kubectl logs --tail=500 -n sim deploy/sim-realtime
+```
+
+Redact any secrets before sharing.

helm/sim/.claude/skills/sim-helm/references/values-model.md

@@ -0,0 +1,149 @@
+# The values.yaml Mental Model
+
+The Sim chart splits configuration across **four** layers. Understanding which layer owns which key is the difference between a working install and a five-hour debugging session.
+
+## The four layers
+
+```
+┌───────────────────────────────────────────────────────────────────────────┐
+│ Layer 1: app.env / realtime.env                                            │
+│   → Written to a chart-managed Kubernetes Secret                           │
+│   → Mounted on pods via envFrom: secretRef                                 │
+│   → Use for: anything sensitive OR anything that varies per-environment   │
+│   → Examples: BETTER_AUTH_SECRET, NEXT_PUBLIC_APP_URL, OPENAI_API_KEY     │
+└───────────────────────────────────────────────────────────────────────────┘
+                                  │
+                                  │ env: (inline) overrides envFrom (Secret)
+                                  ▼
+┌───────────────────────────────────────────────────────────────────────────┐
+│ Layer 2: app.envDefaults / realtime.envDefaults                            │
+│   → Rendered as inline env: on the Deployment                              │
+│   → SKIPPED for any key already set in app.env (or realtime.env)          │
+│   → Use for: operational tunables and safe fallback defaults              │
+│   → Examples: NODE_ENV=production, RATE_LIMIT_*, IVM_*, brand defaults    │
+└───────────────────────────────────────────────────────────────────────────┘
+                                  │
+                                  │ chart-computed values are always inline
+                                  ▼
+┌───────────────────────────────────────────────────────────────────────────┐
+│ Layer 3: chart-computed (inline env: on the Deployment)                    │
+│   → DATABASE_URL, SOCKET_SERVER_URL, OLLAMA_URL                           │
+│   → Derived from postgresql.* / externalDatabase.* / service.* values     │
+│   → CANNOT be overridden via app.env — chart filters them out             │
+└───────────────────────────────────────────────────────────────────────────┘
+                                  │
+                                  │ extraEnvVars appends at the end
+                                  ▼
+┌───────────────────────────────────────────────────────────────────────────┐
+│ Layer 4: extraEnvVars (escape hatch)                                       │
+│   → Raw env: list appended after everything else                          │
+│   → Use for: things the chart doesn't model (valueFrom: configMapKeyRef,  │
+│              custom fieldRef, downward API)                                │
+└───────────────────────────────────────────────────────────────────────────┘
+```
+
+## Why this layering exists
+
+**ESO compatibility.** When `externalSecrets.enabled=true`, the chart-managed Secret is **not rendered** — ESO renders one instead. Anything in Layer 1 must be mapped via `remoteRefs.app.<KEY>` or it's silently missing. Layers 2–4 are unaffected by ESO.
+
+**Override precedence.** Values set in `app.env` (Layer 1 overrides) win over `envDefaults` (Layer 2) — so users who already had operational tunables in `app.env` continue to work.
+
+## Where keys live — the canonical list
+
+The exhaustive list of keys per layer lives in `helm/sim/values.yaml`. Read the file directly when you need to know "is X a Secret key or a tunable?" — it's grouped by layer with comments.
+
+| Concern | Layer | Example keys |
+|---|---|---|
+| Auth secrets | 1 (app.env) | `BETTER_AUTH_SECRET`, `ENCRYPTION_KEY`, `INTERNAL_API_SECRET`, `CRON_SECRET`, `API_ENCRYPTION_KEY` |
+| Provider API keys | 1 (app.env) | `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_CLIENT_SECRET`, etc. |
+| Per-environment URLs | 1 (app.env) | `NEXT_PUBLIC_APP_URL`, `BETTER_AUTH_URL`, `NEXT_PUBLIC_SOCKET_URL` |
+| Feature flags | 1 (app.env) | `ACCESS_CONTROL_ENABLED`, `ORGANIZATIONS_ENABLED`, `SSO_ENABLED`, all `NEXT_PUBLIC_*_ENABLED` |
+| Brand / whitelabel | 1 (app.env) | `NEXT_PUBLIC_BRAND_NAME`, `NEXT_PUBLIC_BRAND_LOGO_URL`, etc. |
+| Operational defaults | 2 (envDefaults) | `NODE_ENV=production`, `EMAIL_VERIFICATION_ENABLED=false`, `VERTEX_LOCATION=us-central1`, `NEXT_PUBLIC_SUPPORT_EMAIL=help@sim.ai` |
+| Rate limits | 2 (envDefaults) | `RATE_LIMIT_WINDOW_MS`, `RATE_LIMIT_FREE_SYNC`, etc. |
+| Execution timeouts | 2 (envDefaults) | `EXECUTION_TIMEOUT_FREE`, `EXECUTION_TIMEOUT_PRO`, etc. |
+| IVM pool / quotas | 2 (envDefaults) | `IVM_POOL_SIZE`, `IVM_MAX_CONCURRENT`, `IVM_MAX_PER_WORKER`, etc. |
+| Connection strings | 3 (chart-computed) | `DATABASE_URL`, `SOCKET_SERVER_URL`, `OLLAMA_URL` |
+| Custom downward API / configMapKeyRef | 4 (extraEnvVars) | anything that needs `valueFrom:` |
+
+## Common authoring patterns
+
+### "I want to set OPENAI_API_KEY for the app"
+
+```yaml
+app:
+  env:
+    OPENAI_API_KEY: "sk-..."  # ends up in the app Secret, mounted via envFrom
+```
+
+For ESO:
+
+```yaml
+externalSecrets:
+  remoteRefs:
+    app:
+      OPENAI_API_KEY: sim/providers/openai-api-key
+```
+
+### "I want to bump the rate limit"
+
+```yaml
+app:
+  envDefaults:
+    RATE_LIMIT_FREE_SYNC: "100"   # overrides the chart's default of 50
+```
+
+Or override it as a regular env var (also valid — Layer 1 wins over Layer 2):
+
+```yaml
+app:
+  env:
+    RATE_LIMIT_FREE_SYNC: "100"
+```
+
+Prefer Layer 2 for non-sensitive tunables — keeps the Secret lean and ESO mapping minimal.
+
+### "I want to set my production app URL"
+
+```yaml
+app:
+  env:
+    NEXT_PUBLIC_APP_URL: "https://sim.example.com"
+    BETTER_AUTH_URL: "https://sim.example.com"
+```
+
+This is the right answer for any clustered deploy. The chart's default is `http://localhost:3000` (Layer 2) — fine for kind/minikube, broken for production. The realtime Deployment also reads these via the shared Secret.
+
+### "I want to inject a value from another ConfigMap"
+
+Use Layer 4:
+
+```yaml
+extraEnvVars:
+  - name: SOME_VALUE
+    valueFrom:
+      configMapKeyRef:
+        name: my-config
+        key: some-key
+```
+
+### "I want to change DATABASE_URL"
+
+You can't override it directly — it's Layer 3, chart-computed. Set the inputs instead:
+
+- For chart-bundled Postgres: edit `postgresql.auth.username`, `.database`, `.port`
+- For external Postgres: enable `externalDatabase.enabled=true` and set `host`, `port`, `username`, `database`, `sslMode`
+
+The chart will compose `DATABASE_URL` from those values.
+
+## Override precedence — the actual K8s rule
+
+When a key exists in both inline `env:` and `envFrom:`:
+
+```
+container.env (Layer 2, 3, 4) WINS over container.envFrom (Layer 1)
+```
+
+This is the Kubernetes spec, not a chart quirk. It's the reason for the override-skip logic in Layer 2: if you set `NEXT_PUBLIC_APP_URL` in Layer 1 (the Secret), the chart **must not** inline the same key in Layer 2 — otherwise the localhost default would mask your prod URL on the realtime pod (which mounts the same shared Secret as the app pod).
+
+The chart handles this correctly for both `app` and `realtime` Deployments. If you ever see a stale value on a pod, check whether the same key is set in **both** `app.env` and `realtime.env` — the merge order in `secrets-app.yaml` makes `app.env` authoritative for shared keys.