chore: import upstream snapshot with attribution

2026-07-13 12:40:49 +08:00
commit 98e40dac97
2013 changed files with 268743 additions and 0 deletions
@@ -0,0 +1,330 @@
+# Multi-process happy-server
+
+How handy-server runs across multiple Kubernetes replicas: socket distribution,
+room-based RPC routing, broadcast fan-out, daemon lifecycle, and what happens
+during the messy cases (pod kill, brief reconnect, network partition).
+
+For the shorter high-level control-flow doc, see `realtime-sync-and-rpc.md`.
+
+> **Status:** the code in this doc is on `main` but `handy.yaml` ships
+> `replicas: 1`. Flipping prod to multi-replica is a separate decision.
+
+## TL;DR
+
+handy-server uses the **Socket.IO Redis streams adapter** to forward
+`io.to(...).emit(...)` between replicas through a single Redis stream. RPC
+routing (web → daemon) goes through **Socket.IO rooms** named
+`rpc:<userId>:<method>`. The server resolves the daemon socket via
+`io.in(room).fetchSockets()` (the cluster-adapter primitive that works
+cross-replica), and sends the request to a single RemoteSocket. There is **no
+Redis key, no TTL, no Lua-CAS cleanup, no keep-alive refresh path** —
+membership is standard Socket.IO room state, cleaned up automatically on
+disconnect.
+
+If the daemon is briefly offline at call time (k8s pod cycling, transient
+network drop), the server **waits up to 10 seconds** for it to reappear before
+failing. If the daemon is in flight when its socket dies, a **presence poll**
+aborts the call within ~1 second instead of waiting the full 30s
+emit-with-ack timeout.
+
+`connectionStateRecovery` is **commented out** in `socket.ts`. The streams
+adapter supports it (verified working) but we ship parity with the
+pre-multi-process behavior first; clients still do a full REST re-fetch on
+every reconnect via `apiSocket.onReconnected`.
+
+## What an rpc-call does (control flow)
+
+```
+rpc-call from web client
+.
+├── input validation
+│   └── method name → invalid → callback({ok:false, error:'Invalid parameters'})
+│
+├── 1. resolve target via cluster adapter
+│   └── fetchRoomSockets(io, 'rpc:<userId>:<method>')
+│       ├── io.in(room).timeout(500ms).fetchSockets()
+│       ├── on success → returns [...]
+│       └── on failure (peer replica unresponsive, fast adapter timeout)
+│           └── log + return [] (treat as "nobody here")
+│       │
+│       ├── returns [target] → go to step 2
+│       └── returns []      → go to wait-for-reconnect
+│
+├── wait-for-reconnect grace (only when no target found)
+│   └── waitForRoomMember(io, room, 10_000ms)
+│       └── poll every 200ms via fetchRoomSockets:
+│           ├── room gained a member → return [target]
+│           └── deadline reached     → return []
+│       │
+│       ├── grace produced [target] → go to step 2
+│       └── grace produced []
+│           └── callback({ok:false, error:'RPC method not available'})
+│
+├── 2. sanity checks on resolved target
+│   ├── multiple sockets in room → log warn, use first
+│   └── target.id === socket.id  → callback({ok:false, error:'same socket'})
+│
+├── 3. fire emit + race a presence poll
+│   ├── ackPromise = target.timeout(30_000).emitWithAck('rpc-request', ...)
+│   │   (cluster adapter routes cross-replica via Redis stream)
+│   │
+│   └── presencePoll = while (alive)
+│       └── sleep 1s, fetchRoomSockets again
+│           ├── target still in room → keep watching
+│           └── target absent       → throw 'RPC target disconnected'
+│
+├── Promise.race(ackPromise, presencePoll)
+│   ├── ackPromise resolves → callback({ok:true, result})
+│   ├── ackPromise throws (timeout / err) → callback({ok:false, error: msg})
+│   └── presencePoll throws → callback({ok:false, error:'RPC target disconnected'})
+│
+└── finally
+    └── presenceAlive = false  (stops the poll cleanly on success or failure)
+```
+
+## What a daemon does (lifecycle)
+
+```
+daemon (machine-scoped or session-scoped)
+.
+├── connect to handy-server
+│   └── server: socket.handshake.auth.token → auth.verifyToken
+│       └── attaches rpcHandler / *UpdateHandler / etc
+│
+├── emit('rpc-register', { method })
+│   └── server: socket.join('rpc:<userId>:<method>')
+│       └── ack: emit('rpc-registered', { method })
+│       (Socket.IO room state, NO Redis key, NO TTL)
+│
+├── on('rpc-request', (data, cb) => …)
+│   └── handler runs, cb(result) returns the value via the cluster adapter
+│
+├── disconnect (any reason)
+│   └── Socket.IO automatically removes the socket from all rooms
+│       (cluster adapter syncs via heartbeat; no manual cleanup needed)
+│
+└── auto-reconnect
+    └── on 'connect': re-emit rpc-register
+        (the only client-side responsibility)
+```
+
+## What a broadcast does (event emission)
+
+```
+eventRouter.emitUpdate / emitEphemeral
+.
+└── io.to(rooms).emit('update' | 'ephemeral', payload)
+    ├── streams adapter: XADD on the 'socket.io' Redis stream
+    │   (MAXLEN ~ 50000, auto-trimmed by Redis)
+    └── every replica's XREAD loop picks up the entry
+        └── delivers to its local sockets that match the room set
+            (sockets that disconnected before the emit miss it; client
+             falls through to apiSocket onReconnected → REST refetch)
+```
+
+Rooms used by `eventRouter`:
+
+```
+.
+├── user:<userId>                              all of a user's sockets
+├── user:<userId>:user-scoped                  only the web/desktop clients
+├── user:<userId>:session:<sessionId>          session-scoped subscribers
+└── user:<userId>:machine:<machineId>          one specific machine
+```
+
+## Where the code lives
+
+```
+.
+├── packages/happy-server/sources/app/
+│   ├── api/socket.ts                      io.Server setup, attaches the
+│   │                                       streams adapter when REDIS_URL
+│   │                                       is set, commented-out
+│   │                                       connectionStateRecovery
+│   ├── api/socket/rpcHandler.ts           the entire RPC routing layer
+│   │                                       (~180 lines, single code path)
+│   ├── api/socket/machineUpdateHandler.ts no longer touches RPC state
+│   ├── api/socket/sessionUpdateHandler.ts no longer touches RPC state
+│   └── events/eventRouter.ts              broadcast emission via rooms
+│
+└── packages/happy-server/deploy/handy.yaml  k8s Deployment + Service
+                                             (replicas: 1 in this PR)
+```
+
+## What was wrong before (the four bugs)
+
+The previous attempt stored RPC routing state as `rpc:user:<u>:method:<m>` →
+socketId Redis keys with a 60-second TTL refreshed by `machine-alive` /
+`session-alive` heartbeats. This had three killer bugs (smoking gun was #3):
+
+```
+.
+├── #1  In-flight RPC eats the full 30s timeout when the target pod dies
+│       io.to(deadSocketId).emitWithAck() has no fast-fail.
+│       FIX: presence poll aborts within ~1s
+│
+├── #2  Reconnect race
+│       Between the daemon's disconnect cleanup and re-register, ~5–7% of
+│       cross-pod RPCs fail with either "method not available" (key
+│       deleted) or "target not reachable" (key still pointed at dead
+│       socketId).
+│       FIX: atomic socket.join / auto-leave on disconnect, no race window
+│
+├── #3  Silent TTL expiry
+│       Daemon stays connected, registration vanishes after 60s if the
+│       keep-alive event was missed for any reason. Daemon never knows;
+│       stays broken until reconnect.
+│       FIX: no TTL exists anymore
+│
+└── #4  Streams adapter "unbounded growth"
+        FALSE ALARM. The adapter trims with MAXLEN ~ on every XADD. Capped
+        at ~50k entries. Crossing this off the list.
+```
+
+The full postmortem with reproduction commands is at
+`deploy/integration-tests/POSTMORTEM.md`.
+
+## How we tested it
+
+Local minikube with a 2-replica handy-server, Redis, Postgres, exposed as a
+real `LoadBalancer` service via `minikube tunnel`. All harnesses live in
+`deploy/integration-tests/`.
+
+```
+.
+├── test-rpc-cross-replica.mjs   steady-state cross-pod RPC
+│                                 (50 parallel + 20 sequential)
+├── test-multiprocess.mjs        broadcast fan-out + pod-kill recovery
+├── hammer.mjs <scenario>        pod-kill-mid-rpc, reconnect-storm,
+│                                 ttl-expiry, brief-disconnect,
+│                                 long-disconnect
+├── network-loss.mjs             long-running RPC loop with summary,
+│                                 usable with iptables blackouts
+├── missed-events.mjs            brief disconnect → triggered broadcast →
+│                                 reconnect; verifies missed-events
+│                                 behavior matches main (lost from socket,
+│                                 recovered via REST refetch)
+├── probe-rpc.mjs                direct rpc-register sanity probe +
+│                                 Redis key inspector
+├── probe-fetchsockets.mjs       fetchSockets latency probe
+├── POSTMORTEM.md                full bug-by-bug
+└── ../local.sh                  bring up the whole minikube stack
+```
+
+To bring up the test environment from scratch:
+
+```bash
+deploy/local.sh                                        # provisions stack
+kubectl get pods -l app=handy-server                   # confirm 2 replicas
+kubectl patch svc handy-server -p '{"spec":{"type":"LoadBalancer"}}'
+minikube tunnel &                                      # exposes :3000
+node deploy/integration-tests/test-rpc-cross-replica.mjs
+```
+
+Final gauntlet result against the fix:
+
+```
+.
+├── steady-state cross-pod RPC          50/50 + 20/20 ✅ (after ~5s warmup)
+├── pod-kill-mid-rpc                    1612ms fast-fail ✅ (was 30000ms)
+├── brief-disconnect                    SUCCESS in 2011ms ✅
+├── long-disconnect                     bounded 10542ms ✅ (10s grace + ~0.5s)
+├── ttl-expiry (smoking gun)            ALL 5 calls pass through +75s ✅
+├── reconnect-storm (5 cycles)          96–97% success ✅ (only inherent
+│                                         in-flight failures, ~3%)
+├── broadcast multi-process             20/20 fan-out, 5/5 unaffected ✅
+├── network-loss 60s loop               85/85 zero failures ✅
+└── missed-events parity                event lost via socket, in DB,
+                                        recovered=undefined ✅ (matches main)
+```
+
+## Tunable constants
+
+```
+RPC_RECONNECT_GRACE_MS        10_000   wait-for-reconnect window (2× heartbeat)
+RPC_RECONNECT_POLL_MS            200   poll cadence inside the grace
+RPC_PRESENCE_POLL_MS           1_000   presence-poll cadence during in-flight
+RPC_PRESENCE_FETCH_TIMEOUT_MS    500   per-call cross-replica fetchSockets cap
+RPC_CALL_TIMEOUT_MS           30_000   upper bound on emitWithAck — same as main
+                                       (no support for >30s RPCs in either)
+```
+
+## Adapter details and limits worth knowing
+
+```
+.
+├── streams adapter discovery
+│   ~5s after a pod starts, the adapter's heartbeat exchange means
+│   cross-replica fetchSockets() may not see all rooms. First few RPCs
+│   immediately after a fresh rollout can hit the wait-for-reconnect
+│   grace; we sized RPC_RECONNECT_GRACE_MS at 10s to cover 2 heartbeat
+│   cycles.
+│
+├── MAXLEN ~ 50000
+│   configured in socket.ts. Auto-trims on every XADD, no cleanup needed.
+│
+├── fetchSockets() cross-replica
+│   defaults to a 5-second timeout per request. We pass timeout(500) for
+│   our presence polls so a single unresponsive replica doesn't stall
+│   every poll for 5s.
+│
+├── emitWithAck from a RemoteSocket
+│   works cross-replica through the cluster adapter (the streams adapter
+│   inherits ClusterAdapterWithHeartbeat which implements BROADCAST_ACK
+│   and FETCH_SOCKETS_RESPONSE).
+│
+└── multiple sockets in the same RPC room
+    shouldn't happen in practice (one daemon per machine, one method
+    registration). If it does, we log a warn and pick targets[0]. Same
+    blast radius as the previous Redis last-write-wins behavior.
+```
+
+## What we still don't do (intentional, deferred)
+
+```
+.
+├── connectionStateRecovery
+│   Commented out in socket.ts. Enabling it would let brief disconnects
+│   skip the heavy REST refetch (events replay through the streams
+│   adapter via restoreSession). Verified working — not shipped to
+│   preserve parity with main on this dimension.
+│
+├── In-flight RPC continuity across daemon reconnect
+│   Coupled to the above. With connectionStateRecovery enabled AND a
+│   recovery-aware presence poll (i.e. "wait N seconds for the same
+│   socketId to come back before failing"), an in-flight RPC could
+│   survive a brief network blip on the daemon: the daemon's handler
+│   keeps running, the ack packet sits in the client's sendBuffer,
+│   reconnect flushes it, the caller gets its result. Today the presence
+│   poll fast-fails the call as soon as the room is empty, which kills
+│   this case. Out of scope for this PR.
+│
+├── User-affinity routing at the LB
+│   Cross-pod RPC overhead is ~3–6ms via the streams adapter. JWT-aware
+│   routing (Envoy / Istio / nginx-lua) would be a bigger infra change
+│   than the fix itself. Tracked as future-work.
+│
+├── UI "reconnecting…" indicator
+│   Server now waits 10s for daemons. Client doesn't yet show that wait
+│   in the UI. apiSocket-side change, separate from this PR.
+│
+├── Tuning the adapter discovery window
+│   5s is the streams adapter's default heartbeatInterval. Lowering it
+│   would reduce the fresh-pod-startup race but increase Redis chatter.
+│
+└── Long-running RPCs (> 30s)
+    Not supported on either main or this PR. Bash command in the CLI has
+    its own 30s cap that races dead-even with the server's 30s emit
+    timeout. Bumping requires both server and (possibly added) client
+    timeouts.
+```
+
+## Reference
+
+- Socket.IO rooms: <https://socket.io/docs/v4/rooms/>
+- `fetchSockets()`: <https://socket.io/docs/v4/server-api/#serverfetchsockets>
+- Broadcasting events: <https://socket.io/docs/v4/broadcasting-events/>
+- Memory usage: <https://socket.io/docs/v4/memory-usage/>
+- Streams adapter source: <https://github.com/socketio/socket.io-redis-streams-adapter>
+- Connection state recovery: <https://socket.io/docs/v4/connection-state-recovery>
+- Discussion #5062 (broadcast emitWithAck waits for all): <https://github.com/socketio/socket.io/discussions/5062>