# Multi-process happy-server How handy-server runs across multiple Kubernetes replicas: socket distribution, room-based RPC routing, broadcast fan-out, daemon lifecycle, and what happens during the messy cases (pod kill, brief reconnect, network partition). For the shorter high-level control-flow doc, see `realtime-sync-and-rpc.md`. > **Status:** the code in this doc is on `main` but `handy.yaml` ships > `replicas: 1`. Flipping prod to multi-replica is a separate decision. ## TL;DR handy-server uses the **Socket.IO Redis streams adapter** to forward `io.to(...).emit(...)` between replicas through a single Redis stream. RPC routing (web → daemon) goes through **Socket.IO rooms** named `rpc::`. The server resolves the daemon socket via `io.in(room).fetchSockets()` (the cluster-adapter primitive that works cross-replica), and sends the request to a single RemoteSocket. There is **no Redis key, no TTL, no Lua-CAS cleanup, no keep-alive refresh path** — membership is standard Socket.IO room state, cleaned up automatically on disconnect. If the daemon is briefly offline at call time (k8s pod cycling, transient network drop), the server **waits up to 10 seconds** for it to reappear before failing. If the daemon is in flight when its socket dies, a **presence poll** aborts the call within ~1 second instead of waiting the full 30s emit-with-ack timeout. `connectionStateRecovery` is **commented out** in `socket.ts`. The streams adapter supports it (verified working) but we ship parity with the pre-multi-process behavior first; clients still do a full REST re-fetch on every reconnect via `apiSocket.onReconnected`. ## What an rpc-call does (control flow) ``` rpc-call from web client . ├── input validation │ └── method name → invalid → callback({ok:false, error:'Invalid parameters'}) │ ├── 1. resolve target via cluster adapter │ └── fetchRoomSockets(io, 'rpc::') │ ├── io.in(room).timeout(500ms).fetchSockets() │ ├── on success → returns [...] │ └── on failure (peer replica unresponsive, fast adapter timeout) │ └── log + return [] (treat as "nobody here") │ │ │ ├── returns [target] → go to step 2 │ └── returns [] → go to wait-for-reconnect │ ├── wait-for-reconnect grace (only when no target found) │ └── waitForRoomMember(io, room, 10_000ms) │ └── poll every 200ms via fetchRoomSockets: │ ├── room gained a member → return [target] │ └── deadline reached → return [] │ │ │ ├── grace produced [target] → go to step 2 │ └── grace produced [] │ └── callback({ok:false, error:'RPC method not available'}) │ ├── 2. sanity checks on resolved target │ ├── multiple sockets in room → log warn, use first │ └── target.id === socket.id → callback({ok:false, error:'same socket'}) │ ├── 3. fire emit + race a presence poll │ ├── ackPromise = target.timeout(30_000).emitWithAck('rpc-request', ...) │ │ (cluster adapter routes cross-replica via Redis stream) │ │ │ └── presencePoll = while (alive) │ └── sleep 1s, fetchRoomSockets again │ ├── target still in room → keep watching │ └── target absent → throw 'RPC target disconnected' │ ├── Promise.race(ackPromise, presencePoll) │ ├── ackPromise resolves → callback({ok:true, result}) │ ├── ackPromise throws (timeout / err) → callback({ok:false, error: msg}) │ └── presencePoll throws → callback({ok:false, error:'RPC target disconnected'}) │ └── finally └── presenceAlive = false (stops the poll cleanly on success or failure) ``` ## What a daemon does (lifecycle) ``` daemon (machine-scoped or session-scoped) . ├── connect to handy-server │ └── server: socket.handshake.auth.token → auth.verifyToken │ └── attaches rpcHandler / *UpdateHandler / etc │ ├── emit('rpc-register', { method }) │ └── server: socket.join('rpc::') │ └── ack: emit('rpc-registered', { method }) │ (Socket.IO room state, NO Redis key, NO TTL) │ ├── on('rpc-request', (data, cb) => …) │ └── handler runs, cb(result) returns the value via the cluster adapter │ ├── disconnect (any reason) │ └── Socket.IO automatically removes the socket from all rooms │ (cluster adapter syncs via heartbeat; no manual cleanup needed) │ └── auto-reconnect └── on 'connect': re-emit rpc-register (the only client-side responsibility) ``` ## What a broadcast does (event emission) ``` eventRouter.emitUpdate / emitEphemeral . └── io.to(rooms).emit('update' | 'ephemeral', payload) ├── streams adapter: XADD on the 'socket.io' Redis stream │ (MAXLEN ~ 50000, auto-trimmed by Redis) └── every replica's XREAD loop picks up the entry └── delivers to its local sockets that match the room set (sockets that disconnected before the emit miss it; client falls through to apiSocket onReconnected → REST refetch) ``` Rooms used by `eventRouter`: ``` . ├── user: all of a user's sockets ├── user::user-scoped only the web/desktop clients ├── user::session: session-scoped subscribers └── user::machine: one specific machine ``` ## Where the code lives ``` . ├── packages/happy-server/sources/app/ │ ├── api/socket.ts io.Server setup, attaches the │ │ streams adapter when REDIS_URL │ │ is set, commented-out │ │ connectionStateRecovery │ ├── api/socket/rpcHandler.ts the entire RPC routing layer │ │ (~180 lines, single code path) │ ├── api/socket/machineUpdateHandler.ts no longer touches RPC state │ ├── api/socket/sessionUpdateHandler.ts no longer touches RPC state │ └── events/eventRouter.ts broadcast emission via rooms │ └── packages/happy-server/deploy/handy.yaml k8s Deployment + Service (replicas: 1 in this PR) ``` ## What was wrong before (the four bugs) The previous attempt stored RPC routing state as `rpc:user::method:` → socketId Redis keys with a 60-second TTL refreshed by `machine-alive` / `session-alive` heartbeats. This had three killer bugs (smoking gun was #3): ``` . ├── #1 In-flight RPC eats the full 30s timeout when the target pod dies │ io.to(deadSocketId).emitWithAck() has no fast-fail. │ FIX: presence poll aborts within ~1s │ ├── #2 Reconnect race │ Between the daemon's disconnect cleanup and re-register, ~5–7% of │ cross-pod RPCs fail with either "method not available" (key │ deleted) or "target not reachable" (key still pointed at dead │ socketId). │ FIX: atomic socket.join / auto-leave on disconnect, no race window │ ├── #3 Silent TTL expiry │ Daemon stays connected, registration vanishes after 60s if the │ keep-alive event was missed for any reason. Daemon never knows; │ stays broken until reconnect. │ FIX: no TTL exists anymore │ └── #4 Streams adapter "unbounded growth" FALSE ALARM. The adapter trims with MAXLEN ~ on every XADD. Capped at ~50k entries. Crossing this off the list. ``` The full postmortem with reproduction commands is at `deploy/integration-tests/POSTMORTEM.md`. ## How we tested it Local minikube with a 2-replica handy-server, Redis, Postgres, exposed as a real `LoadBalancer` service via `minikube tunnel`. All harnesses live in `deploy/integration-tests/`. ``` . ├── test-rpc-cross-replica.mjs steady-state cross-pod RPC │ (50 parallel + 20 sequential) ├── test-multiprocess.mjs broadcast fan-out + pod-kill recovery ├── hammer.mjs pod-kill-mid-rpc, reconnect-storm, │ ttl-expiry, brief-disconnect, │ long-disconnect ├── network-loss.mjs long-running RPC loop with summary, │ usable with iptables blackouts ├── missed-events.mjs brief disconnect → triggered broadcast → │ reconnect; verifies missed-events │ behavior matches main (lost from socket, │ recovered via REST refetch) ├── probe-rpc.mjs direct rpc-register sanity probe + │ Redis key inspector ├── probe-fetchsockets.mjs fetchSockets latency probe ├── POSTMORTEM.md full bug-by-bug └── ../local.sh bring up the whole minikube stack ``` To bring up the test environment from scratch: ```bash deploy/local.sh # provisions stack kubectl get pods -l app=handy-server # confirm 2 replicas kubectl patch svc handy-server -p '{"spec":{"type":"LoadBalancer"}}' minikube tunnel & # exposes :3000 node deploy/integration-tests/test-rpc-cross-replica.mjs ``` Final gauntlet result against the fix: ``` . ├── steady-state cross-pod RPC 50/50 + 20/20 ✅ (after ~5s warmup) ├── pod-kill-mid-rpc 1612ms fast-fail ✅ (was 30000ms) ├── brief-disconnect SUCCESS in 2011ms ✅ ├── long-disconnect bounded 10542ms ✅ (10s grace + ~0.5s) ├── ttl-expiry (smoking gun) ALL 5 calls pass through +75s ✅ ├── reconnect-storm (5 cycles) 96–97% success ✅ (only inherent │ in-flight failures, ~3%) ├── broadcast multi-process 20/20 fan-out, 5/5 unaffected ✅ ├── network-loss 60s loop 85/85 zero failures ✅ └── missed-events parity event lost via socket, in DB, recovered=undefined ✅ (matches main) ``` ## Tunable constants ``` RPC_RECONNECT_GRACE_MS 10_000 wait-for-reconnect window (2× heartbeat) RPC_RECONNECT_POLL_MS 200 poll cadence inside the grace RPC_PRESENCE_POLL_MS 1_000 presence-poll cadence during in-flight RPC_PRESENCE_FETCH_TIMEOUT_MS 500 per-call cross-replica fetchSockets cap RPC_CALL_TIMEOUT_MS 30_000 upper bound on emitWithAck — same as main (no support for >30s RPCs in either) ``` ## Adapter details and limits worth knowing ``` . ├── streams adapter discovery │ ~5s after a pod starts, the adapter's heartbeat exchange means │ cross-replica fetchSockets() may not see all rooms. First few RPCs │ immediately after a fresh rollout can hit the wait-for-reconnect │ grace; we sized RPC_RECONNECT_GRACE_MS at 10s to cover 2 heartbeat │ cycles. │ ├── MAXLEN ~ 50000 │ configured in socket.ts. Auto-trims on every XADD, no cleanup needed. │ ├── fetchSockets() cross-replica │ defaults to a 5-second timeout per request. We pass timeout(500) for │ our presence polls so a single unresponsive replica doesn't stall │ every poll for 5s. │ ├── emitWithAck from a RemoteSocket │ works cross-replica through the cluster adapter (the streams adapter │ inherits ClusterAdapterWithHeartbeat which implements BROADCAST_ACK │ and FETCH_SOCKETS_RESPONSE). │ └── multiple sockets in the same RPC room shouldn't happen in practice (one daemon per machine, one method registration). If it does, we log a warn and pick targets[0]. Same blast radius as the previous Redis last-write-wins behavior. ``` ## What we still don't do (intentional, deferred) ``` . ├── connectionStateRecovery │ Commented out in socket.ts. Enabling it would let brief disconnects │ skip the heavy REST refetch (events replay through the streams │ adapter via restoreSession). Verified working — not shipped to │ preserve parity with main on this dimension. │ ├── In-flight RPC continuity across daemon reconnect │ Coupled to the above. With connectionStateRecovery enabled AND a │ recovery-aware presence poll (i.e. "wait N seconds for the same │ socketId to come back before failing"), an in-flight RPC could │ survive a brief network blip on the daemon: the daemon's handler │ keeps running, the ack packet sits in the client's sendBuffer, │ reconnect flushes it, the caller gets its result. Today the presence │ poll fast-fails the call as soon as the room is empty, which kills │ this case. Out of scope for this PR. │ ├── User-affinity routing at the LB │ Cross-pod RPC overhead is ~3–6ms via the streams adapter. JWT-aware │ routing (Envoy / Istio / nginx-lua) would be a bigger infra change │ than the fix itself. Tracked as future-work. │ ├── UI "reconnecting…" indicator │ Server now waits 10s for daemons. Client doesn't yet show that wait │ in the UI. apiSocket-side change, separate from this PR. │ ├── Tuning the adapter discovery window │ 5s is the streams adapter's default heartbeatInterval. Lowering it │ would reduce the fresh-pod-startup race but increase Redis chatter. │ └── Long-running RPCs (> 30s) Not supported on either main or this PR. Bash command in the CLI has its own 30s cap that races dead-even with the server's 30s emit timeout. Bumping requires both server and (possibly added) client timeouts. ``` ## Reference - Socket.IO rooms: - `fetchSockets()`: - Broadcasting events: - Memory usage: - Streams adapter source: - Connection state recovery: - Discussion #5062 (broadcast emitWithAck waits for all):