stablyai--orca/docs/reference/ssh-typing-latency-under-relay-load.md

SSH Typing Latency Under Relay Load

Date: 2026-07-06

Symptom

Typing in an SSH-host terminal can become extremely slow (hundreds of ms to seconds per echoed keystroke) while other SSH work is active — file previews, source-control refresh, search. A quiet SSH shell stays snappy, which made the reports look unreproducible.

This is distinct from the local warm-switch lag documented in terminal-switch-typing-lag-investigation.md (daemon listSessions() snapshot cost — already fixed; TerminalHost.listSessions() is now metadata-only via getAppliedSize()).

Root Cause

The relay and the Electron client share ONE ordered SSH channel for all JSON-RPC traffic: PTY input/output, file streams, git responses, search, port scans. Two mechanisms turned bulk traffic into typing latency:

1. Relay-side head-of-line blocking (primary). The fs.readFileStream pump (src/relay/fs-handler-file-read.ts) wrote every 256KB chunk (~340KB framed) into the relay's stdout as fast as local disk reads completed, ignoring the write() === false backpressure signal. A 10MB preview enqueued ~13.6MB into the pipe at once; a pty.data echo emitted mid-stream queued behind ALL of it. At 2MB/s WAN that is multiple seconds of echo delay per open file. The reproduced measurement: 4,195,592 bytes (the entire remaining 3MB test file, framed) queued ahead of one echo frame.

2. Client-side O(n²) frame buffering (secondary). FrameDecoder.feed() in both src/main/ssh/relay-protocol.ts and src/relay/protocol.ts did Buffer.concat([buffered, chunk]) per data event, re-copying the whole backlog for every ~32KB TCP chunk — ~2MB of memcpy per 340KB frame, ~80MB per 10MB file — on the Electron main thread, delaying pty:write IPC dispatch and echo delivery to the renderer.

Fix

• Bulk lane with sink backpressure (RelayDispatcher.notifyBulk): fs.streamChunk frames are serialized per client and each send waits for the sink's drain when write() returns false. Interactive frames (pty.data) still use plain notify() and are admitted immediately — because the bulk lane keeps the outbound buffer at ~1 frame, an echo jumps ahead of every not-yet-admitted chunk. Sinks (relay.ts stdout + Unix socket clients) surface write()'s boolean and a one-shot waitWriteDrain; every death path (EPIPE, stdin end, detach, dispose) flushes parked waiters so a pump can never hang on a dead pipe.

• Credit-based flow control (fs.streamAck): the client requests streams with flowControl: 'ack' and acks each processed chunk; the relay caps unacked chunks at STREAM_ACK_WINDOW_CHUNKS = 4 (~1MB raw). This bounds in-flight bulk bytes even past the relay's own pipe (sshd/TCP buffers) and paces the relay's base64/JSON encode loop so incoming keystroke frames get event-loop turns. A parked pump wakes on ack, cancel, release, client detach, and a 1s staleness recheck.

• Cross-version compatibility: old client + new relay → no flowControl param → legacy unpaced pump (still drain-bounded, which is transparent). New client + old relay → fs.streamAck is an unknown notification and is silently ignored. STREAM_CHUNK_SIZE (256KB) is intentionally unchanged — both sides bake it into chunk offset math, so changing it would corrupt cross-version streams.

• FrameDecoder rewrite (both mirrored protocol files): chunk-list buffering; each frame is assembled exactly once (one copy) instead of re-concatenating the backlog per feed.

What Stays Fast / Unchanged

• Relay PTY output batching (interactive echo fast path in src/relay/pty-handler.ts) is untouched.
• No listSessions() / provider inventory was added anywhere near typing, focus, switch, resume, resize, or render paths.
• Remote file streaming correctness: per-chunk length checks, chunk-count and byte-count invariants, cancel paths, and the 12MB round-trip integration test all pass unchanged.

Regression Coverage

• src/relay/fs-stream-pty-echo-backpressure.integration.test.ts — real mux ↔ dispatcher ↔ FsHandler over a congestible in-memory pipe; asserts deterministic BYTE bounds (not wall-clock): echo queues behind < 2 framed chunks when congested (pre-fix: whole file), ack window caps in-flight chunks, legacy no-ack clients still get complete streams.
• src/relay/fs-handler-stream.test.ts — pump parks at the ack window, resumes per ack, and releases its file handle when cancelled while parked.
• src/relay/dispatcher.test.ts — notifyBulk semantics: drain gating, per-client targeting, dispose releases parked senders, interactive notify() not gated behind a stalled bulk lane.
• src/main/ssh/relay-protocol.test.ts — decoder byte-at-a-time boundary straddling, oversized-frame resync across odd chunk sizes, and a no-Buffer.concat-during-feed guard that locks out the O(n²) shape.
• tests/e2e/ssh-docker-relay-perf.spec.ts — new "busy relay" scenario: types while two 8MB remote file-read loops and a git.status loop run; budgets median < 500ms, worst < 2000ms. (On loopback Docker the pre-fix HOL is only tens of ms — the in-process byte-bound test is the authoritative red/green; the e2e guards the end-to-end wiring.)

Residual Gaps

• Large single-frame responses: git.* responses (stdout capped at MAX_GIT_BUFFER = 10MB → ~13MB framed) and other request/response results are one atomic frame on the wire; a huge diff can still delay an echo by its own transfer time. Fixing this requires response chunking (a protocol change); bounded by the 16MB MAX_MESSAGE_SIZE.
• Outbound direction: a large fs.writeFile request frame (client → relay) can queue ahead of keystroke pty.data frames on channel.stdin. Same shape, opposite direction, rarer trigger (saving a large remote file while typing).
• pty.ackData flow control for PTY output remains unenforced (src/relay/pty-handler.ts — "not yet enforced"); PTY output floods are already paced by the relay's 8ms batch/16KB slice scheduler.