chore: import upstream snapshot with attribution

helm/sim/.claude/skills/sim-helm/SKILL.md

@@ -0,0 +1,154 @@
+---
+name: sim-helm
+description: Install, upgrade, and operate the Sim Helm chart on Kubernetes. Covers install path selection (inline / existingSecret / External Secrets Operator), required secret generation, the values.yaml mental model (env vs envDefaults vs Secret), and common failure triage. Invoke when a user asks about deploying Sim to a cluster, authoring a Sim values.yaml, debugging a Sim pod that won't start, upgrading a Sim release, or wiring Sim into a secret manager.
+license: Apache-2.0
+---
+
+# Sim Helm Chart — Operations Skill
+
+This skill helps an agent deploy and operate the **Sim** Helm chart at `helm/sim/` in the [simstudioai/sim](https://github.com/simstudioai/sim) repository. Use it when the user is installing, upgrading, troubleshooting, or authoring values for the Sim chart.
+
+The skill is **diagnostic-first**: capture context, classify the situation, load only the references that apply, then act. Do not dump the README at the user. Do not invent values that are not in their current state.
+
+---
+
+## Workflow — follow in order
+
+### 1. Capture context
+
+Before recommending anything, ask (or infer from the conversation) all of these. **Never skip this step.** A wrong assumption here corrupts every downstream step.
+
+| Question | Why it matters |
+|---|---|
+| Cluster: EKS / GKE / AKS / OpenShift / kind / other? | Storage class, ingress class, identity provider differ |
+| Secret strategy: inline `--set`, pre-existing K8s Secret, or External Secrets Operator (ESO)? | The chart has three distinct code paths |
+| Postgres: chart-bundled, or external (RDS / Cloud SQL / Azure DB)? | Different value blocks (`postgresql.*` vs `externalDatabase.*`) |
+| Public-facing? Ingress class? TLS? | `ingress.enabled`, `ingress.className`, cert-manager wiring |
+| HA? (target replicas) | Drives `autoscaling.enabled`, `app.replicaCount`, PDB activation |
+| Existing values.yaml the user is editing? | Always read it before proposing a diff — never write blind |
+
+If the user has a `values.yaml`, read it. If they don't, ask before writing one.
+
+### 2. Diagnose
+
+Map the user's request to one of these categories and load the matching reference(s):
+
+| Situation | Reference |
+|---|---|
+| User wants to install for the first time | `references/install-paths.md` then `references/secrets.md` |
+| User needs to generate the required secrets | `references/secrets.md` |
+| User asks "what does this value do" / wants to author values.yaml | `references/values-model.md` |
+| Pod won't start, error message, `CrashLoopBackOff`, image pull error, ingress not routing | `references/troubleshooting.md` |
+| User asks about ESO / Vault / AWS Secrets Manager / Azure Key Vault / GCP Secret Manager | `references/install-paths.md` (ESO section) |
+| User asks "is X production-ready" / autoscaling / network policy / security context | Read the README's "Production checklist" section directly — no separate reference |
+
+Load **only** what the situation requires. Loading every reference burns tokens and produces vague answers.
+
+### 3. Propose
+
+When proposing values changes:
+
+- Show the **minimal diff** against the user's current values.yaml. Don't rewrite the file.
+- Name the **risk** (e.g., "this puts the secret in `helm get values` output — fine for dev, not for prod").
+- Name the **rollback** (e.g., "if this breaks, `helm rollback sim 1` reverts").
+- Cite the canonical source (`helm/sim/values.yaml` line numbers, README section, or this skill's reference file).
+
+### 4. Validate before applying
+
+Always run these before telling the user to `helm install` / `helm upgrade`:
+
+```bash
+# Schema + value validation
+helm lint helm/sim --values <user-values>.yaml
+
+# Render full manifest set to catch template errors
+helm template sim helm/sim --values <user-values>.yaml > /tmp/render.yaml
+
+# For upgrades, render against the live release first
+helm upgrade --dry-run sim helm/sim --values <user-values>.yaml
+```
+
+If lint or template fails, fix the values — do not work around chart validation. The chart's `fail` statements exist to catch misconfigurations that would otherwise surface as `CrashLoopBackOff` at runtime.
+
+### 5. Deliver
+
+Every recommendation should include:
+
+- The exact command(s) to run
+- A one-line summary of what will change
+- The success signal (e.g., "`kubectl rollout status deploy/sim-app` returns Ready")
+- The rollback command if something breaks
+
+---
+
+## Quick reference — the three secret modes
+
+| Mode | When | Code path |
+|---|---|---|
+| **Inline (`--set`)** | Dev / kind / dry-run only. Values leak into `helm get values`. | `app.env.<KEY>: "..."` |
+| **Pre-existing Secret** | GitOps with Sealed Secrets / SOPS, or hand-managed Secrets. Chart references a Secret you create. | `app.secrets.existingSecret.enabled: true` + `.name` |
+| **External Secrets Operator (recommended for prod)** | Vault, AWS SM, Azure KV, GCP SM. Chart renders an `ExternalSecret` that ESO syncs. | `externalSecrets.enabled: true` + `secretStoreRef` + `remoteRefs.app.<KEY>` |
+
+These modes are **mutually exclusive** for the app Secret. ESO takes precedence over inline. `existingSecret` takes precedence over inline. The chart **fails template rendering** when ESO is enabled and a required key (`BETTER_AUTH_SECRET`, `ENCRYPTION_KEY`, `INTERNAL_API_SECRET`, plus `CRON_SECRET` when cronjobs are enabled) is neither in `app.env` nor mapped in `remoteRefs.app` — see `references/install-paths.md`.
+
+---
+
+## Quick reference — the four required secrets
+
+| Key | Generate with | Notes |
+|---|---|---|
+| `BETTER_AUTH_SECRET` | `openssl rand -hex 32` | Session signing |
+| `ENCRYPTION_KEY` | `openssl rand -hex 32` | App-level encryption |
+| `INTERNAL_API_SECRET` | `openssl rand -hex 32` | Service-to-service auth (app ↔ realtime) |
+| `CRON_SECRET` | `openssl rand -hex 32` | Required iff `cronjobs.enabled=true` (default true) |
+
+Optional but commonly needed:
+
+| Key | Generate with | Notes |
+|---|---|---|
+| `API_ENCRYPTION_KEY` | `openssl rand -hex 32` | Must be **exactly 64 hex chars**. Required to encrypt user API keys at rest. |
+| `postgresql.auth.password` | `openssl rand -base64 24 \| tr -d '/+='` | Only if using chart-bundled Postgres. Must match `^[a-zA-Z0-9._-]+$` for DATABASE_URL compatibility. |
+
+See `references/secrets.md` for storage patterns and rotation guidance.
+
+---
+
+## Rules of engagement
+
+These are non-negotiable. Violating any of these has burned users in the past.
+
+1. **Never recommend `--set` for production secrets.** They land in `helm get values` and Helm release history. Direct users to `existingSecret` or ESO.
+2. **Never set `image.tag: latest`.** The chart defaults to `Chart.AppVersion` for a reason — reproducible rollouts. If the user pinned `latest`, push back.
+3. **Never edit chart templates to work around a `fail` statement.** The validation exists because a misconfiguration would otherwise surface as a runtime CrashLoopBackOff with cryptic env errors.
+4. **Never drop `automountServiceAccountToken: false`** unless the workload genuinely needs in-cluster API access (Sim's app/realtime/postgres pods do not).
+5. **Never `kubectl delete sts` without `--cascade=orphan`** on a live Postgres. It deletes the pods and PVCs.
+6. **Never tell a user "the chart works on your cluster" without `helm lint` + `helm template` against their values.** Static reading is not validation.
+7. **Always confirm before `helm uninstall` in a shared namespace.** PVCs survive but other namespace resources may not.
+
+---
+
+## When the user is stuck and you can't diagnose
+
+Get logs from every component in parallel. This single block answers ~80% of "it's broken" questions:
+
+```bash
+kubectl --namespace <ns> get pods,events --sort-by='.lastTimestamp'
+kubectl --namespace <ns> logs deploy/sim-app --tail=200
+kubectl --namespace <ns> logs deploy/sim-realtime --tail=200
+kubectl --namespace <ns> logs sts/sim-postgresql --tail=200
+kubectl --namespace <ns> logs job/sim-migrations --tail=200 2>/dev/null
+kubectl --namespace <ns> describe pod -l app.kubernetes.io/name=sim
+```
+
+Then map the symptom to `references/troubleshooting.md`.
+
+---
+
+## What this skill does **not** cover
+
+- Sim application configuration beyond env vars (provider keys, knowledge base setup, etc.) — that's the Sim app docs at https://docs.sim.ai
+- Kubernetes cluster setup (creating an EKS cluster, installing ingress-nginx, etc.) — that's cloud-provider docs
+- Authoring new chart templates — that's `helm/sim/templates/_helpers.tpl` and the chart's own contributor docs
+- Running Sim outside Kubernetes (Docker Compose, bare-metal) — see the root `README.md`
+
+If the user's question falls outside this scope, say so and point them at the right doc.

helm/sim/.claude/skills/sim-helm/references/install-paths.md

@@ -0,0 +1,192 @@
+# Install Path Selection
+
+Three mutually-exclusive paths for the app Secret. Pick exactly one. The chart enforces this at template time.
+
+## Decision tree
+
+```
+Is this a production install?
+├── No (dev / kind / minikube / dry-run)
+│     → Inline `--set` is fine. Skip to "Path A".
+│
+└── Yes
+      │
+      Do you already manage secrets with Vault / AWS Secrets Manager /
+      Azure Key Vault / GCP Secret Manager / 1Password Connect?
+      │
+      ├── Yes → External Secrets Operator. Path C.
+      │
+      └── No
+            │
+            Do you use GitOps with Sealed Secrets, SOPS, or
+            hand-managed Kubernetes Secrets?
+            │
+            ├── Yes → Pre-existing Secret. Path B.
+            │
+            └── No → Install ESO and go to Path C.
+                    (Don't skip to inline `--set` for prod —
+                    secrets land in `helm get values` and release history.)
+```
+
+---
+
+## Path A — Inline `--set` (dev only)
+
+```bash
+helm install sim ./helm/sim \
+  --namespace sim --create-namespace \
+  --set app.env.BETTER_AUTH_SECRET=$(openssl rand -hex 32) \
+  --set app.env.ENCRYPTION_KEY=$(openssl rand -hex 32) \
+  --set app.env.INTERNAL_API_SECRET=$(openssl rand -hex 32) \
+  --set app.env.CRON_SECRET=$(openssl rand -hex 32) \
+  --set postgresql.auth.password=$(openssl rand -base64 24 | tr -d '/+=')
+```
+
+The chart generates a `Secret` named `<release>-app-secrets` containing every non-empty key from `app.env` + `realtime.env`. Both `app` and `realtime` Deployments mount it via `envFrom`.
+
+**Risks:**
+- Secrets are visible in `helm get values <release>` and `helm history <release>`.
+- Anyone with read access to the release's ConfigMap (`sh.helm.release.v1.<release>.v<N>`) can recover the secrets — they're stored base64-encoded inside.
+
+---
+
+## Path B — Pre-existing Kubernetes Secret
+
+Create the Secret first, then point the chart at it.
+
+```bash
+kubectl create namespace sim
+kubectl create secret generic sim-app-secrets --namespace sim \
+  --from-literal=BETTER_AUTH_SECRET=$(openssl rand -hex 32) \
+  --from-literal=ENCRYPTION_KEY=$(openssl rand -hex 32) \
+  --from-literal=INTERNAL_API_SECRET=$(openssl rand -hex 32) \
+  --from-literal=CRON_SECRET=$(openssl rand -hex 32)
+
+kubectl create secret generic sim-postgres-secret --namespace sim \
+  --from-literal=POSTGRES_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=')
+```
+
+```yaml
+# values.yaml
+app:
+  secrets:
+    existingSecret:
+      enabled: true
+      name: sim-app-secrets
+
+postgresql:
+  auth:
+    existingSecret:
+      enabled: true
+      name: sim-postgres-secret
+      passwordKey: POSTGRES_PASSWORD
+```
+
+**The chart cannot introspect your Secret.** If you forget a required key, the pod will fail at runtime with `CreateContainerConfigError: secret key "X" not found`. The required keys are: `BETTER_AUTH_SECRET`, `ENCRYPTION_KEY`, `INTERNAL_API_SECRET`, plus `CRON_SECRET` when cronjobs are enabled.
+
+For GitOps (Sealed Secrets / SOPS), seal/encrypt the Secret YAML before committing — never commit a plain `kubectl create secret` output.
+
+---
+
+## Path C — External Secrets Operator (production recommended)
+
+ESO syncs from your existing secret store (Vault, AWS SM, Azure KV, GCP SM, etc.) into a Kubernetes Secret on a refresh interval. The chart renders the `ExternalSecret` resource; ESO does the syncing.
+
+### Prerequisites
+
+1. Install ESO once per cluster:
+   ```bash
+   helm repo add external-secrets https://charts.external-secrets.io
+   helm install external-secrets external-secrets/external-secrets \
+     -n external-secrets --create-namespace
+   ```
+2. Create a `ClusterSecretStore` (or namespace-scoped `SecretStore`) that points at your secret manager. ESO's docs cover the auth wiring for each provider.
+
+### Values
+
+```yaml
+externalSecrets:
+  enabled: true
+  apiVersion: v1beta1     # v1beta1 works on ESO >= 0.7. Bump to v1 only on ESO >= 0.17.
+  refreshInterval: 1h
+  secretStoreRef:
+    name: my-cluster-secret-store
+    kind: ClusterSecretStore     # or SecretStore for namespace-scoped
+  remoteRefs:
+    app:
+      BETTER_AUTH_SECRET: sim/app/better-auth-secret
+      ENCRYPTION_KEY: sim/app/encryption-key
+      INTERNAL_API_SECRET: sim/app/internal-api-secret
+      CRON_SECRET: sim/app/cron-secret     # required iff cronjobs.enabled
+      # Optional but commonly mapped:
+      API_ENCRYPTION_KEY: sim/app/api-encryption-key
+      OPENAI_API_KEY: sim/providers/openai
+    postgresql:
+      password: sim/postgresql/password    # required if postgresql.enabled
+    externalDatabase:
+      password: sim/postgresql/password    # required if externalDatabase.enabled
+
+# Leave app.env empty (or only set non-secret values like NEXT_PUBLIC_APP_URL).
+app:
+  env: {}
+```
+
+### Fail-fast behavior
+
+The chart will refuse to render if:
+
+- `externalSecrets.enabled=true` and any of `BETTER_AUTH_SECRET`, `ENCRYPTION_KEY`, `INTERNAL_API_SECRET` (or `CRON_SECRET` when cronjobs are enabled) is **neither** set in `app.env` **nor** mapped in `remoteRefs.app`. Error message names the missing key.
+- A key is set in `app.env` with a non-empty value but not mapped in `remoteRefs.app` (would be silently dropped from the rendered Secret).
+
+These checks catch the "renders cleanly, CrashLoopBackOffs at runtime" failure mode that plagued earlier chart versions.
+
+### Remote ref shapes
+
+Each `remoteRefs.app.<KEY>` value can be either:
+
+```yaml
+# Shorthand — just the path/key in the store
+BETTER_AUTH_SECRET: sim/app/better-auth-secret
+```
+
+```yaml
+# Full form — pass any field ESO supports
+BETTER_AUTH_SECRET:
+  key: sim/app/better-auth-secret
+  property: value          # for stores that return JSON
+  version: "v3"            # pin a specific version
+  decodingStrategy: Base64 # for base64-stored values
+```
+
+---
+
+## Cross-cutting: things that are NOT secrets
+
+Operational tunables (rate limits, timeouts, IVM pool size, branding) live in `app.envDefaults` and `realtime.envDefaults`. They're rendered as **inline `env:`** on the Deployment, not written to the Secret. See `values-model.md` for the full mental model.
+
+Don't try to push these into ESO — they're not sensitive, they'd just bloat the secret store.
+
+---
+
+## Verifying your choice
+
+After `helm install`:
+
+```bash
+# What Secret will the pods mount?
+helm template sim helm/sim -f my-values.yaml | grep -A2 "envFrom:"
+
+# For ESO: did the ExternalSecret render?
+helm template sim helm/sim -f my-values.yaml | grep -B1 -A10 "kind: ExternalSecret"
+
+# For existingSecret: is your pre-created Secret referenced?
+helm template sim helm/sim -f my-values.yaml | grep -E "name: .*-app-secrets"
+```
+
+For ESO, after `helm install`, verify the sync:
+
+```bash
+kubectl get externalsecret -n sim
+kubectl describe externalsecret <release>-app-secrets -n sim
+# Status should show 'SecretSynced=True'
+```

helm/sim/.claude/skills/sim-helm/references/secrets.md

@@ -0,0 +1,88 @@
+# Secret Generation
+
+The Sim chart requires four cryptographic secrets at install time. Generate them once and store them in your chosen path (see `install-paths.md`). Never reuse these across environments.
+
+## Generate all four at once
+
+```bash
+export BETTER_AUTH_SECRET=$(openssl rand -hex 32)
+export ENCRYPTION_KEY=$(openssl rand -hex 32)
+export INTERNAL_API_SECRET=$(openssl rand -hex 32)
+export CRON_SECRET=$(openssl rand -hex 32)
+
+# Optional but commonly needed:
+export API_ENCRYPTION_KEY=$(openssl rand -hex 32)       # MUST be exactly 64 hex chars
+export POSTGRES_PASSWORD=$(openssl rand -base64 24 | tr -d '/+=')  # if using chart-bundled Postgres
+```
+
+## What each secret does
+
+| Key | Purpose | Length | Rotation impact |
+|---|---|---|---|
+| `BETTER_AUTH_SECRET` | Signs user session JWTs (Better Auth) | 32 bytes = 64 hex chars | Rotating invalidates all active sessions — users must re-login |
+| `ENCRYPTION_KEY` | App-level encryption for sensitive fields | 32 bytes = 64 hex chars | Rotating breaks decryption of existing data — requires migration |
+| `INTERNAL_API_SECRET` | Shared auth between `sim-app` ↔ `sim-realtime` pods | 32 bytes = 64 hex chars | Both deployments must roll together — temporary realtime errors during the rollout |
+| `CRON_SECRET` | Authenticates scheduled CronJob pods to the app | 32 bytes = 64 hex chars | Rotating just needs `helm upgrade`; next cron run uses the new value |
+| `API_ENCRYPTION_KEY` (optional) | Encrypts user-stored API keys (OpenAI tokens, etc.) at rest in Postgres | **Exactly 64 hex chars** (the app rejects other lengths) | Without it, keys are stored plain. Once set, never rotate without a migration |
+| `POSTGRES_PASSWORD` (chart-bundled Postgres only) | Postgres superuser password | Any length ≥ 12 chars matching `^[a-zA-Z0-9._-]+$` | Requires Postgres pod restart + app rollout |
+
+The `^[a-zA-Z0-9._-]+$` constraint on the Postgres password exists because the chart embeds the password into `DATABASE_URL` without URL-encoding. The `tr -d '/+='` strips the three problematic characters from `openssl rand -base64` output. The chart enforces this regex at template time.
+
+## Storage by path
+
+### Path A (inline `--set`)
+
+Pass each on the command line — see `install-paths.md` Path A.
+
+### Path B (pre-existing Kubernetes Secret)
+
+```bash
+kubectl create namespace sim
+kubectl create secret generic sim-app-secrets --namespace sim \
+  --from-literal=BETTER_AUTH_SECRET=$BETTER_AUTH_SECRET \
+  --from-literal=ENCRYPTION_KEY=$ENCRYPTION_KEY \
+  --from-literal=INTERNAL_API_SECRET=$INTERNAL_API_SECRET \
+  --from-literal=CRON_SECRET=$CRON_SECRET \
+  --from-literal=API_ENCRYPTION_KEY=$API_ENCRYPTION_KEY
+
+kubectl create secret generic sim-postgres-secret --namespace sim \
+  --from-literal=POSTGRES_PASSWORD=$POSTGRES_PASSWORD
+```
+
+For GitOps, run the `kubectl create secret ... --dry-run=client -o yaml` and pipe through `kubeseal` (Sealed Secrets) or `sops` before committing.
+
+### Path C (External Secrets Operator)
+
+Push the generated values into your secret manager first. Example for AWS Secrets Manager:
+
+```bash
+aws secretsmanager create-secret --name sim/app/better-auth-secret --secret-string "$BETTER_AUTH_SECRET"
+aws secretsmanager create-secret --name sim/app/encryption-key     --secret-string "$ENCRYPTION_KEY"
+aws secretsmanager create-secret --name sim/app/internal-api-secret --secret-string "$INTERNAL_API_SECRET"
+aws secretsmanager create-secret --name sim/app/cron-secret        --secret-string "$CRON_SECRET"
+aws secretsmanager create-secret --name sim/app/api-encryption-key --secret-string "$API_ENCRYPTION_KEY"
+aws secretsmanager create-secret --name sim/postgresql/password    --secret-string "$POSTGRES_PASSWORD"
+```
+
+Then map the paths in `externalSecrets.remoteRefs.app` (see `install-paths.md` Path C).
+
+## Rotation
+
+Sim doesn't have built-in rotation hooks. The procedure is:
+
+1. Generate a new value, store it.
+2. `helm upgrade` (or let ESO pick up the change on its next refresh).
+3. Restart the affected workloads to force re-read of `envFrom`:
+   ```bash
+   kubectl rollout restart deploy/sim-app deploy/sim-realtime -n sim
+   ```
+4. For `BETTER_AUTH_SECRET`: expect a wave of `401`s as old sessions invalidate.
+5. For `ENCRYPTION_KEY` / `API_ENCRYPTION_KEY`: **do not rotate** without an explicit data migration. Existing ciphertext becomes undecryptable.
+
+## What NOT to do
+
+- **Don't reuse the same secret across dev/staging/prod.** A leak in one tier compromises all.
+- **Don't commit secrets to git, even in private repos.** Use sealed-secrets / SOPS / ESO.
+- **Don't paste secrets into Slack, Discord, GitHub issues, or screenshots.** Treat them like database passwords.
+- **Don't store secrets in `values.yaml` files committed to git.** That's worse than `--set` — values files persist forever in history.
+- **Don't generate secrets with weak entropy.** No `date | md5`, no `password123`, no developer's birthday. `openssl rand` or `/dev/urandom` only.

helm/sim/.claude/skills/sim-helm/references/troubleshooting.md

@@ -0,0 +1,183 @@
+# Troubleshooting
+
+Map symptom → root cause → fix. Always run the diagnostic block first, then match.
+
+## Diagnostic block (run this first)
+
+```bash
+NS=sim  # adjust to your namespace
+kubectl --namespace $NS get pods,events --sort-by='.lastTimestamp'
+kubectl --namespace $NS describe pod -l app.kubernetes.io/instance=sim
+kubectl --namespace $NS logs deploy/sim-app --tail=200
+kubectl --namespace $NS logs deploy/sim-realtime --tail=200
+kubectl --namespace $NS logs sts/sim-postgresql --tail=200
+kubectl --namespace $NS logs job/sim-migrations --tail=200 2>/dev/null || true
+```
+
+---
+
+## `Error: execution error at (sim/templates/...): app.env.BETTER_AUTH_SECRET is required for production deployment`
+
+**Cause:** `helm install` / `helm upgrade` ran without the required secrets set.
+
+**Fix:** Generate and pass the four required secrets (see `references/secrets.md`). For production, switch to `existingSecret` or ESO instead of `--set` (see `references/install-paths.md`).
+
+---
+
+## `Error: execution error ...: Required key 'X' is missing: externalSecrets.enabled=true but the key is neither set in app.env nor mapped in externalSecrets.remoteRefs.app`
+
+**Cause:** ESO is enabled but one of the required keys (`BETTER_AUTH_SECRET`, `ENCRYPTION_KEY`, `INTERNAL_API_SECRET`, or `CRON_SECRET` when cronjobs are enabled) isn't mapped via `remoteRefs.app`. The chart fails fast at template time to avoid CrashLoopBackOff later.
+
+**Fix:** Add the mapping:
+
+```yaml
+externalSecrets:
+  remoteRefs:
+    app:
+      <KEY>: path/in/your/secret/store
+```
+
+Or, if you really don't need cronjobs, set `cronjobs.enabled=false` to drop the `CRON_SECRET` requirement.
+
+---
+
+## `Error: execution error ...: Key 'X' is set in app.env but externalSecrets.enabled=true and externalSecrets.remoteRefs.app.X is not configured`
+
+**Cause:** ESO is enabled. The chart-managed Secret is not rendered. A key set in `app.env` would be silently dropped — pods would start with the wrong (missing) value.
+
+**Fix:** Either map the key via `remoteRefs.app.X` so ESO syncs it, OR remove the key from `app.env` if you don't need it.
+
+---
+
+## App pods stuck in `CrashLoopBackOff`
+
+Get the logs first:
+
+```bash
+kubectl logs --namespace sim deploy/sim-app --tail 200
+```
+
+Match the error:
+
+| Log line | Cause | Fix |
+|---|---|---|
+| `Invalid env: ... NEXT_PUBLIC_APP_URL: Invalid url` | URL field set to empty string or invalid format | Set `app.env.NEXT_PUBLIC_APP_URL` to a valid URL — `https://sim.example.com` in prod, `http://localhost:3000` in dev |
+| `getaddrinfo ENOTFOUND ... -postgresql` / `connect ECONNREFUSED` | App can't reach Postgres | Check `kubectl get pod -l app.kubernetes.io/name=postgresql` is `Running`; check `postgresql.auth.password` matches the password in the Secret |
+| `password authentication failed for user "sim"` | Postgres password rotated but app pod wasn't restarted, OR password contains URL-unsafe chars | `kubectl rollout restart deploy/sim-app -n sim`; regenerate password with `openssl rand -base64 24 \| tr -d '/+='` |
+| `BETTER_AUTH_SECRET is missing` / `INTERNAL_API_SECRET is required` | Required env var not present in the Secret | Verify with `kubectl get secret sim-app-secrets -o jsonpath='{.data}' \| jq 'keys'`; if missing, fix your secret strategy |
+| `Migration failed` or app starts before migration | Migration Job hasn't completed | `kubectl logs job/sim-migrations -n sim`; rerun with `kubectl delete job/sim-migrations && helm upgrade ...` |
+
+---
+
+## Image pull errors (`ErrImagePull` / `ImagePullBackOff`)
+
+```bash
+kubectl describe pod -l app.kubernetes.io/name=sim -n sim | grep -A5 "Failed\|Warning"
+```
+
+| Cause | Fix |
+|---|---|
+| Private registry, no pull secret | Set `global.imagePullSecrets: [{name: my-regcred}]` and create the regcred Secret: `kubectl create secret docker-registry my-regcred --docker-server=... --docker-username=... --docker-password=...` |
+| Image tag doesn't exist in the registry | `helm get values sim`, check the rendered `image.tag`; correct it or fall back to `Chart.AppVersion` |
+| Air-gapped cluster | Mirror the image to your internal registry, set `global.imageRegistry=my-registry.example.com` |
+
+---
+
+## Postgres pod `Pending`
+
+```bash
+kubectl describe pvc --namespace sim
+```
+
+Always one of:
+
+| `Events` says | Cause | Fix |
+|---|---|---|
+| `no persistent volumes available for this claim and no storage class is set` | No default StorageClass | Set `global.storageClass: <your-class>` or annotate one as default |
+| `Failed to provision volume with StorageClass "X"` | No PV provisioner installed | Install one (`local-path-provisioner` for kind, EBS CSI for EKS, PD CSI for GKE, Azure Disk CSI for AKS) |
+| `only ReadWriteOnce access modes are supported` | StorageClass doesn't support RWO | Pick a different `global.storageClass` |
+| `pod has unbound immediate PersistentVolumeClaims` and no events | StorageClass uses `WaitForFirstConsumer` and pod isn't schedulable | Check pod's `nodeSelector` / `tolerations` against your nodes |
+
+---
+
+## Ingress not routing
+
+```bash
+kubectl get ingress -n sim
+kubectl describe ingress -n sim
+```
+
+| Cause | Fix |
+|---|---|
+| No `ADDRESS` in `kubectl get ingress` | Ingress controller not installed — install `ingress-nginx`, AWS LBC, GCP LB controller, etc. |
+| `ingressClassName` doesn't match installed controller | `kubectl get ingressclass` to list installed classes, set `ingress.className` to match |
+| Address is set but DNS resolves to wrong IP | `dig <your-host>` — point DNS at the ingress controller's external IP / LoadBalancer / CNAME |
+| TLS cert errors | If using cert-manager, check `kubectl describe certificate -n sim`; verify `ingress.tls.issuerRef` |
+| `503 Service Unavailable` | Ingress routing is fine but app pod isn't `Ready` — go back to the diagnostic block |
+
+---
+
+## CronJob pods fail with `CreateContainerConfigError: couldn't find key CRON_SECRET in Secret`
+
+**Cause:** `cronjobs.enabled=true` (the default) but `CRON_SECRET` isn't in the app Secret. Two paths:
+
+1. Inline mode: `app.env.CRON_SECRET=""` — the chart will fail at template time. If you somehow got past that, regenerate and set it.
+2. Existing-Secret mode: your pre-created Secret doesn't include `CRON_SECRET`. Add it:
+   ```bash
+   kubectl patch secret sim-app-secrets -n sim --type='json' \
+     -p='[{"op":"add","path":"/data/CRON_SECRET","value":"'$(openssl rand -hex 32 | base64)'"}]'
+   ```
+3. ESO mode: missing `remoteRefs.app.CRON_SECRET` mapping. Add it.
+
+Or set `cronjobs.enabled=false` if you don't need scheduled jobs.
+
+---
+
+## `app.kubernetes.io/managed-by: Helm` collisions when upgrading
+
+You installed once with `--name foo`, then tried to install again with `--name bar` into the same namespace. Resources collide on labels.
+
+**Fix:** Use distinct namespaces per release, or `helm uninstall foo -n <ns>` first.
+
+---
+
+## Pods get OOMKilled
+
+```bash
+kubectl get events -n sim --field-selector reason=OOMKilling
+```
+
+Bump the relevant resource limit. Defaults:
+
+| Workload | Default request | Default limit |
+|---|---|---|
+| `app` | `1000m` CPU / `4Gi` memory | `2000m` CPU / `8Gi` memory |
+| `realtime` | `250m` CPU / `512Mi` memory | `500m` CPU / `1Gi` memory |
+| `postgresql` | `250m` CPU / `512Mi` memory | `1000m` CPU / `2Gi` memory |
+
+Override in values:
+
+```yaml
+app:
+  resources:
+    requests:
+      memory: 8Gi
+    limits:
+      memory: 16Gi
+```
+
+---
+
+## Logs to grab when filing a support issue
+
+```bash
+helm version
+kubectl version --short
+helm get values sim -n sim --revision $(helm history sim -n sim | tail -1 | awk '{print $1}')
+kubectl get all,pvc,ingress,externalsecret -n sim -o wide
+kubectl describe pods -n sim -l app.kubernetes.io/instance=sim | head -200
+kubectl logs --tail=500 -n sim deploy/sim-app
+kubectl logs --tail=500 -n sim deploy/sim-realtime
+```
+
+Redact any secrets before sharing.

helm/sim/.claude/skills/sim-helm/references/values-model.md

@@ -0,0 +1,149 @@
+# The values.yaml Mental Model
+
+The Sim chart splits configuration across **four** layers. Understanding which layer owns which key is the difference between a working install and a five-hour debugging session.
+
+## The four layers
+
+```
+┌───────────────────────────────────────────────────────────────────────────┐
+│ Layer 1: app.env / realtime.env                                            │
+│   → Written to a chart-managed Kubernetes Secret                           │
+│   → Mounted on pods via envFrom: secretRef                                 │
+│   → Use for: anything sensitive OR anything that varies per-environment   │
+│   → Examples: BETTER_AUTH_SECRET, NEXT_PUBLIC_APP_URL, OPENAI_API_KEY     │
+└───────────────────────────────────────────────────────────────────────────┘
+                                  │
+                                  │ env: (inline) overrides envFrom (Secret)
+                                  ▼
+┌───────────────────────────────────────────────────────────────────────────┐
+│ Layer 2: app.envDefaults / realtime.envDefaults                            │
+│   → Rendered as inline env: on the Deployment                              │
+│   → SKIPPED for any key already set in app.env (or realtime.env)          │
+│   → Use for: operational tunables and safe fallback defaults              │
+│   → Examples: NODE_ENV=production, RATE_LIMIT_*, IVM_*, brand defaults    │
+└───────────────────────────────────────────────────────────────────────────┘
+                                  │
+                                  │ chart-computed values are always inline
+                                  ▼
+┌───────────────────────────────────────────────────────────────────────────┐
+│ Layer 3: chart-computed (inline env: on the Deployment)                    │
+│   → DATABASE_URL, SOCKET_SERVER_URL, OLLAMA_URL                           │
+│   → Derived from postgresql.* / externalDatabase.* / service.* values     │
+│   → CANNOT be overridden via app.env — chart filters them out             │
+└───────────────────────────────────────────────────────────────────────────┘
+                                  │
+                                  │ extraEnvVars appends at the end
+                                  ▼
+┌───────────────────────────────────────────────────────────────────────────┐
+│ Layer 4: extraEnvVars (escape hatch)                                       │
+│   → Raw env: list appended after everything else                          │
+│   → Use for: things the chart doesn't model (valueFrom: configMapKeyRef,  │
+│              custom fieldRef, downward API)                                │
+└───────────────────────────────────────────────────────────────────────────┘
+```
+
+## Why this layering exists
+
+**ESO compatibility.** When `externalSecrets.enabled=true`, the chart-managed Secret is **not rendered** — ESO renders one instead. Anything in Layer 1 must be mapped via `remoteRefs.app.<KEY>` or it's silently missing. Layers 2–4 are unaffected by ESO.
+
+**Override precedence.** Values set in `app.env` (Layer 1 overrides) win over `envDefaults` (Layer 2) — so users who already had operational tunables in `app.env` continue to work.
+
+## Where keys live — the canonical list
+
+The exhaustive list of keys per layer lives in `helm/sim/values.yaml`. Read the file directly when you need to know "is X a Secret key or a tunable?" — it's grouped by layer with comments.
+
+| Concern | Layer | Example keys |
+|---|---|---|
+| Auth secrets | 1 (app.env) | `BETTER_AUTH_SECRET`, `ENCRYPTION_KEY`, `INTERNAL_API_SECRET`, `CRON_SECRET`, `API_ENCRYPTION_KEY` |
+| Provider API keys | 1 (app.env) | `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, `GOOGLE_CLIENT_SECRET`, etc. |
+| Per-environment URLs | 1 (app.env) | `NEXT_PUBLIC_APP_URL`, `BETTER_AUTH_URL`, `NEXT_PUBLIC_SOCKET_URL` |
+| Feature flags | 1 (app.env) | `ACCESS_CONTROL_ENABLED`, `ORGANIZATIONS_ENABLED`, `SSO_ENABLED`, all `NEXT_PUBLIC_*_ENABLED` |
+| Brand / whitelabel | 1 (app.env) | `NEXT_PUBLIC_BRAND_NAME`, `NEXT_PUBLIC_BRAND_LOGO_URL`, etc. |
+| Operational defaults | 2 (envDefaults) | `NODE_ENV=production`, `EMAIL_VERIFICATION_ENABLED=false`, `VERTEX_LOCATION=us-central1`, `NEXT_PUBLIC_SUPPORT_EMAIL=help@sim.ai` |
+| Rate limits | 2 (envDefaults) | `RATE_LIMIT_WINDOW_MS`, `RATE_LIMIT_FREE_SYNC`, etc. |
+| Execution timeouts | 2 (envDefaults) | `EXECUTION_TIMEOUT_FREE`, `EXECUTION_TIMEOUT_PRO`, etc. |
+| IVM pool / quotas | 2 (envDefaults) | `IVM_POOL_SIZE`, `IVM_MAX_CONCURRENT`, `IVM_MAX_PER_WORKER`, etc. |
+| Connection strings | 3 (chart-computed) | `DATABASE_URL`, `SOCKET_SERVER_URL`, `OLLAMA_URL` |
+| Custom downward API / configMapKeyRef | 4 (extraEnvVars) | anything that needs `valueFrom:` |
+
+## Common authoring patterns
+
+### "I want to set OPENAI_API_KEY for the app"
+
+```yaml
+app:
+  env:
+    OPENAI_API_KEY: "sk-..."  # ends up in the app Secret, mounted via envFrom
+```
+
+For ESO:
+
+```yaml
+externalSecrets:
+  remoteRefs:
+    app:
+      OPENAI_API_KEY: sim/providers/openai-api-key
+```
+
+### "I want to bump the rate limit"
+
+```yaml
+app:
+  envDefaults:
+    RATE_LIMIT_FREE_SYNC: "100"   # overrides the chart's default of 50
+```
+
+Or override it as a regular env var (also valid — Layer 1 wins over Layer 2):
+
+```yaml
+app:
+  env:
+    RATE_LIMIT_FREE_SYNC: "100"
+```
+
+Prefer Layer 2 for non-sensitive tunables — keeps the Secret lean and ESO mapping minimal.
+
+### "I want to set my production app URL"
+
+```yaml
+app:
+  env:
+    NEXT_PUBLIC_APP_URL: "https://sim.example.com"
+    BETTER_AUTH_URL: "https://sim.example.com"
+```
+
+This is the right answer for any clustered deploy. The chart's default is `http://localhost:3000` (Layer 2) — fine for kind/minikube, broken for production. The realtime Deployment also reads these via the shared Secret.
+
+### "I want to inject a value from another ConfigMap"
+
+Use Layer 4:
+
+```yaml
+extraEnvVars:
+  - name: SOME_VALUE
+    valueFrom:
+      configMapKeyRef:
+        name: my-config
+        key: some-key
+```
+
+### "I want to change DATABASE_URL"
+
+You can't override it directly — it's Layer 3, chart-computed. Set the inputs instead:
+
+- For chart-bundled Postgres: edit `postgresql.auth.username`, `.database`, `.port`
+- For external Postgres: enable `externalDatabase.enabled=true` and set `host`, `port`, `username`, `database`, `sslMode`
+
+The chart will compose `DATABASE_URL` from those values.
+
+## Override precedence — the actual K8s rule
+
+When a key exists in both inline `env:` and `envFrom:`:
+
+```
+container.env (Layer 2, 3, 4) WINS over container.envFrom (Layer 1)
+```
+
+This is the Kubernetes spec, not a chart quirk. It's the reason for the override-skip logic in Layer 2: if you set `NEXT_PUBLIC_APP_URL` in Layer 1 (the Secret), the chart **must not** inline the same key in Layer 2 — otherwise the localhost default would mask your prod URL on the realtime pod (which mounts the same shared Secret as the app pod).
+
+The chart handles this correctly for both `app` and `realtime` Deployments. If you ever see a stale value on a pod, check whether the same key is set in **both** `app.env` and `realtime.env` — the merge order in `secrets-app.yaml` makes `app.env` authoritative for shared keys.