slopus--happy/packages/happy-server/deploy/integration-tests/AGENTS.md

Integration Tests

Stress tests that run against a real multi-replica Happy server on minikube.

Prerequisites

• minikube
• kubectl
• docker
• node 20+ with socket.io-client, tweetnacl, ioredis available

Start local cluster

bash local.sh

This builds the server image, applies the kustomize overlay (2-replica server + Postgres + Redis + MinIO), runs Prisma migrations, and rolls out the deployment.

After it finishes, open a tunnel to the service:

minikube service handy-server --url

Use the printed URL as SERVER_URL for the tests below. If port-forwarding instead:

kubectl port-forward svc/handy-server 3005:3000
# SERVER_URL=http://127.0.0.1:3005

Run all tests (recommended)

./run-all.sh              # run all tests (cluster must be deployed)
./run-all.sh --deploy     # build, deploy, then run all tests
./run-all.sh --safe-only  # skip pod-killing tests

Handles port-forwards, Redis tunnel, pod recovery between destructive tests, and prints a summary. This is the one-click way.

Automated tests (run via run-all.sh)

These tests work with kubectl port-forward and are run automatically by run-all.sh.

All tests accept SERVER_URL env var (defaults to http://127.0.0.1:3000).

Prod-realistic stress (stream pressure + RPC reliability)

SERVER_URL=http://<tunnel-url> node stress-prod-realistic.mjs [entries_per_sec]

Default rate: 5000 entries/sec.

RPC registration (safe scenarios)

SERVER_URL=http://<tunnel-url> node stress-rpc-registration.mjs <scenario|all>

all runs the port-forward-safe scenarios: fire-and-forget, register-race-timing, reconnect-no-ack, rapid-sessions, ios-session-flow, high-concurrency, cross-replica-3pod.

Dead-daemon detection (per-pod port-forwards)

SERVER_URL=http://<tunnel-url> node test-rpc-dead-daemon.mjs

Kills a pod but sets up per-pod port-forwards so the test survives the pod death. This is the canonical pod-kill test.

Manual tests (NOT port-forward compatible)

These tests kill pods, which breaks the shared kubectl port-forward that run-all.sh and most manual setups use. They must be run individually, ideally through a stable ingress or minikube service URL rather than port-forward.

RPC registration (pod-kill scenarios)

SERVER_URL=http://<tunnel-url> node stress-rpc-registration.mjs rolling-deploy
SERVER_URL=http://<tunnel-url> node stress-rpc-registration.mjs stale-room-cleanup
SERVER_URL=http://<tunnel-url> node stress-rpc-registration.mjs server-rolling-restart

These are excluded from all because they kill pods. test-rpc-dead-daemon.mjs covers the core pod-kill detection case.

Sync degradation and recovery

SERVER_URL=http://<tunnel-url> node stress-sync-degradation.mjs <scenario|all>

Scenarios: full-server-outage, reconnect-connect-err, rpc-after-reconnect, cascading-disruption. ALL scenarios kill pods, so none work through port-forward. test-rpc-dead-daemon.mjs covers the core pod-kill detection scenario.

Useful kubectl commands

# Watch pods
kubectl get pods -w

# Tail server logs
kubectl logs -l app=handy-server --all-containers -f

# Kill a specific pod (for pod-kill scenarios)
kubectl delete pod <pod-name> --grace-period=0

# Restart deployment
kubectl rollout restart deployment/handy-server

# Access Redis CLI
kubectl exec -it happy-redis-0 -- redis-cli

# Access MinIO console
kubectl port-forward svc/happy-minio 9001:9001
# then open http://localhost:9001 (minioadmin/minioadmin)