wehub-resource-sync
116 lines
6.0 KiB
Markdown
116 lines
6.0 KiB
Markdown
# SSH Typing Latency Under Relay Load
|
|
|
|
Date: 2026-07-06
|
|
|
|
## Symptom
|
|
|
|
Typing in an SSH-host terminal can become extremely slow (hundreds of ms to
|
|
seconds per echoed keystroke) while other SSH work is active — file previews,
|
|
source-control refresh, search. A quiet SSH shell stays snappy, which made the
|
|
reports look unreproducible.
|
|
|
|
This is distinct from the local warm-switch lag documented in
|
|
`terminal-switch-typing-lag-investigation.md` (daemon `listSessions()`
|
|
snapshot cost — already fixed; `TerminalHost.listSessions()` is now
|
|
metadata-only via `getAppliedSize()`).
|
|
|
|
## Root Cause
|
|
|
|
The relay and the Electron client share ONE ordered SSH channel for all
|
|
JSON-RPC traffic: PTY input/output, file streams, git responses, search, port
|
|
scans. Two mechanisms turned bulk traffic into typing latency:
|
|
|
|
1. **Relay-side head-of-line blocking (primary).** The `fs.readFileStream`
|
|
pump (`src/relay/fs-handler-file-read.ts`) wrote every 256KB chunk (~340KB
|
|
framed) into the relay's stdout as fast as local disk reads completed,
|
|
ignoring the `write() === false` backpressure signal. A 10MB preview
|
|
enqueued ~13.6MB into the pipe at once; a `pty.data` echo emitted
|
|
mid-stream queued behind ALL of it. At 2MB/s WAN that is multiple seconds
|
|
of echo delay per open file. The reproduced measurement: 4,195,592 bytes
|
|
(the entire remaining 3MB test file, framed) queued ahead of one echo
|
|
frame.
|
|
|
|
2. **Client-side O(n²) frame buffering (secondary).** `FrameDecoder.feed()`
|
|
in both `src/main/ssh/relay-protocol.ts` and `src/relay/protocol.ts` did
|
|
`Buffer.concat([buffered, chunk])` per data event, re-copying the whole
|
|
backlog for every ~32KB TCP chunk — ~2MB of memcpy per 340KB frame,
|
|
~80MB per 10MB file — on the Electron main thread, delaying `pty:write`
|
|
IPC dispatch and echo delivery to the renderer.
|
|
|
|
## Fix
|
|
|
|
- **Bulk lane with sink backpressure** (`RelayDispatcher.notifyBulk`):
|
|
`fs.streamChunk` frames are serialized per client and each send waits for
|
|
the sink's `drain` when `write()` returns `false`. Interactive frames
|
|
(`pty.data`) still use plain `notify()` and are admitted immediately —
|
|
because the bulk lane keeps the outbound buffer at ~1 frame, an echo jumps
|
|
ahead of every not-yet-admitted chunk. Sinks (`relay.ts` stdout + Unix
|
|
socket clients) surface `write()`'s boolean and a one-shot
|
|
`waitWriteDrain`; every death path (EPIPE, stdin end, detach, dispose)
|
|
flushes parked waiters so a pump can never hang on a dead pipe.
|
|
|
|
- **Credit-based flow control** (`fs.streamAck`): the client requests
|
|
streams with `flowControl: 'ack'` and acks each processed chunk; the relay
|
|
caps unacked chunks at `STREAM_ACK_WINDOW_CHUNKS = 4` (~1MB raw). This
|
|
bounds in-flight bulk bytes even past the relay's own pipe (sshd/TCP
|
|
buffers) and paces the relay's base64/JSON encode loop so incoming
|
|
keystroke frames get event-loop turns. A parked pump wakes on ack, cancel,
|
|
release, client detach, and a 1s staleness recheck.
|
|
|
|
- **Cross-version compatibility**: old client + new relay → no
|
|
`flowControl` param → legacy unpaced pump (still drain-bounded, which is
|
|
transparent). New client + old relay → `fs.streamAck` is an unknown
|
|
notification and is silently ignored. `STREAM_CHUNK_SIZE` (256KB) is
|
|
intentionally unchanged — both sides bake it into chunk offset math, so
|
|
changing it would corrupt cross-version streams.
|
|
|
|
- **FrameDecoder rewrite** (both mirrored protocol files): chunk-list
|
|
buffering; each frame is assembled exactly once (one copy) instead of
|
|
re-concatenating the backlog per feed.
|
|
|
|
## What Stays Fast / Unchanged
|
|
|
|
- Relay PTY output batching (interactive echo fast path in
|
|
`src/relay/pty-handler.ts`) is untouched.
|
|
- No `listSessions()` / provider inventory was added anywhere near typing,
|
|
focus, switch, resume, resize, or render paths.
|
|
- Remote file streaming correctness: per-chunk length checks, chunk-count and
|
|
byte-count invariants, cancel paths, and the 12MB round-trip integration
|
|
test all pass unchanged.
|
|
|
|
## Regression Coverage
|
|
|
|
- `src/relay/fs-stream-pty-echo-backpressure.integration.test.ts` — real
|
|
mux ↔ dispatcher ↔ FsHandler over a congestible in-memory pipe; asserts
|
|
deterministic BYTE bounds (not wall-clock): echo queues behind < 2 framed
|
|
chunks when congested (pre-fix: whole file), ack window caps in-flight
|
|
chunks, legacy no-ack clients still get complete streams.
|
|
- `src/relay/fs-handler-stream.test.ts` — pump parks at the ack window,
|
|
resumes per ack, and releases its file handle when cancelled while parked.
|
|
- `src/relay/dispatcher.test.ts` — notifyBulk semantics: drain gating,
|
|
per-client targeting, dispose releases parked senders, interactive
|
|
notify() not gated behind a stalled bulk lane.
|
|
- `src/main/ssh/relay-protocol.test.ts` — decoder byte-at-a-time boundary
|
|
straddling, oversized-frame resync across odd chunk sizes, and a
|
|
no-`Buffer.concat`-during-feed guard that locks out the O(n²) shape.
|
|
- `tests/e2e/ssh-docker-relay-perf.spec.ts` — new "busy relay" scenario:
|
|
types while two 8MB remote file-read loops and a git.status loop run;
|
|
budgets median < 500ms, worst < 2000ms. (On loopback Docker the pre-fix
|
|
HOL is only tens of ms — the in-process byte-bound test is the
|
|
authoritative red/green; the e2e guards the end-to-end wiring.)
|
|
|
|
## Residual Gaps
|
|
|
|
- **Large single-frame responses**: `git.*` responses (stdout capped at
|
|
`MAX_GIT_BUFFER` = 10MB → ~13MB framed) and other request/response results
|
|
are one atomic frame on the wire; a huge diff can still delay an echo by
|
|
its own transfer time. Fixing this requires response chunking (a protocol
|
|
change); bounded by the 16MB `MAX_MESSAGE_SIZE`.
|
|
- **Outbound direction**: a large `fs.writeFile` request frame (client →
|
|
relay) can queue ahead of keystroke `pty.data` frames on `channel.stdin`.
|
|
Same shape, opposite direction, rarer trigger (saving a large remote file
|
|
while typing).
|
|
- `pty.ackData` flow control for PTY output remains unenforced
|
|
(`src/relay/pty-handler.ts` — "not yet enforced"); PTY output floods are
|
|
already paced by the relay's 8ms batch/16KB slice scheduler.
|