elizaos--eliza/packages/docs/ongoing-development/research/05-chat-window-ux.md

Chat window UX: scrolling, gestures, search, mic

Summary

Workstream research for the LifeOps Personal Assistant MVP (GitHub project 15). Chat is the MVP's primary surface: every LifeOps flow (reminders, goals, todos, scheduling, coordination) is driven through the continuous chat overlay, so its scrolling, gesture, search, and voice affordances must be solid for every kind of user — children, adults with ADHD/ADD/Asperger's/autism, neurotypical adults, and elderly people — with no therapy language and no special rails, just a chat window that never fights the hand that drives it.

The audit's headline: the gesture/detent system asked for in the brief is already built, well-documented, and CI-gated (pull-to-maximize landed in #13531; the maximize button is already gone; detent magnetism, springs, and haptics all exist). What is actually missing or broken is narrower: no history pagination at all (hard 80-row render cap over a 200-message server window), a transcript that can silently become horizontally scrollable, message search that exists end-to-end but is unreachable from the primary surface, a mic button that does not pulse while recording on the primary surface, and a stale Playwright suite still asserting buttons #13531 removed. The MVP plan is fix-and-verify, not build: six issues, five of which delete, wire, or repair what already exists.

Current state

Render paths (three, not two)

1. ContinuousChatOverlay — the primary surface, always mounted over every view (packages/ui/src/App.tsx:2513, packages/ui/src/components/shell/ContinuousChatOverlay.tsx:128-157). One infinite thread since #13531 — no conversation switcher, no clear/new-chat.
2. ChatSurface + AssistantOverlay + HomePill — mounted only by ShellFoundationMount for the desktop OS chat-overlay window and the kiosk shell (App.tsx:496, App.tsx:539).
3. ChatView (1,536 lines) — desktop detached chat windows only, via packages/app-core/src/runtime/desktop/AppWindowRenderer.tsx:44 and DetachedShellRoot.tsx:71. Not routed anywhere in the web/mobile shell.

Message text additionally renders through two independent code paths (overlay body renderer vs MessageContent.tsx), pinned by a render-parity contract test (parser-parity.contract.test.ts, render-parity.contract.tsx). Any formatting change must land in both.

Sheet state machine, detents, gestures — built and tested

Single source of truth: ChatMode = "pill" | "input" | "half" | "full" with derived ChatState (CLOSED | INPUT | OPEN_UNDER_HALF | OPEN_HALF_OR_OVER | MAXIMIZED) — ContinuousChatOverlay.tsx:177-193. The interaction constants ARE the spec (all verified in code):

Parameter	Value	Ref (ContinuousChatOverlay.tsx)
HALF detent	0.46 × viewport height	:197
FULL detent	≈0.9 × viewport (inset, under status bar)	:198-206
Pull-to-maximize	peak raw pull ≥ max(0.8×vh, FULL) + 56px over-pull	:207, :3228-3237
Restore-from-maximized	downward pull starting in top 20% of panel	:211, :3878-3892
Detent magnet	release within 64px snaps to detent, else free-rest	:219, :3385-3410
Pill→input morph distance	120px of finger travel; excess flows into thread height	:265, :3158-3183
Sheet spring	stiffness 320 / damping 34 / mass 0.9	:247-252
Pill-open spring	stiffness 300 / damping 26 / mass 0.85	:256-261
Detent haptic	Capacitor light impact per detent cross	:228-246

The live drag writes the flexBasis MotionValue directly (no React re-render per frame, :3985-3995); flicks step one detent; slow drags free-rest with magnetism; pull-down through input→pill reverses the morph under the finger (:3171-3183). The maximize button is already removed — header carries only a Launcher button (:3895-3901), and ContinuousChatOverlay.test.tsx:2308-2309 pins chat-full-maximize / chat-full-clear absent. Pull-to-maximize + top-20% restore are e2e-covered with real CDP touch (__e2e__/run-chat-sheet-e2e.mjs:256-354).

Gesture engine: use-pull-gesture.ts — axis lock at 8px slop, widened 0.8-dominance cone for swipes, rAF-coalesced drag, commit-on-pointercancel for Android (#9943), capture-loss handling for rotation. Weakness: its verticalScrollPriority option (:87-102, built for the transcript, #chat-scroll-web) has zero consumers — dead code since #13531 removed the transcript swipe binding (grep: only definition sites match).

Scrolling — what works and what is missing

Works: the transcript scroller (#continuous-thread, ContinuousChatOverlay.tsx:4025) is touch-pan-y overflow-y-auto overscroll-contain with the iOS -webkit-overflow-scrolling:touch fix and a WebKit-safe bounded-height flex chain (:3966-3995, guarded by run-chat-scroll-web-e2e.mjs on Chromium AND WebKit). Tail-follow / jump-to-latest is the shared useThreadAutoScroll engine (packages/ui/src/hooks/useThreadAutoScroll.ts:1-23, wired at :4126).

Missing — history pagination (req 1). GET /api/conversations/:id/messages returns only the newest 200 (CONVERSATION_MESSAGE_WINDOW, packages/agent/src/api/conversation-routes.ts:1050, :1684-1688); there is no before cursor — only ?around=<id> for search jumps (:1052-1118). The client fetches the window as a full replace (packages/ui/src/state/useDataLoaders.ts:336-368) and the overlay renders at most the last 80 rows (MAX_RENDERED_SHELL_MESSAGES, packages/ui/src/components/shell/shell-state.ts:60). Scrolling up simply ends. runtime.getMemories already supports end+limit+orderBy desc (used by the around-window), so a cursor is a small, store-pushed addition.

Broken — horizontal-scroll "unlock" (req 1). Root cause found statically: the scroller sets overflow-y-auto but not overflow-x-hidden (:4025). Per CSS Overflow, specifying one axis coerces the other from visible to auto — so the moment any child overflows horizontally, the transcript itself becomes a horizontal scroll container. touch-pan-y blocks touch panning, but trackpad/wheel deltas ignore touch-action — on desktop a diagonal two-finger scroll pans the thread sideways. Overflowing children exist today: text/code attachment previews use both-axes overflow-auto with no overscroll-x-contain (MessageAttachments.tsx:831); inline widgets are unaudited for min-width. The other chat path already does this right: chat-thread-layout.tsx:106-107 pairs overflow-x-hidden overflow-y-auto. Designed horizontal scrollers inside the thread (code blocks MessageContent.tsx:572,1242, chips bar TopicChipsBar.tsx:30) are fine — they carry overscroll-x-contain.

Search (req 2) — exists end-to-end, unreachable on the primary surface

• Server: GET /api/conversations/messages/search — SQL ILIKE pushdown scoped to accessible rooms, BM25 ranking, snippets, limit clamp 50 (conversation-routes.ts:1470-1545).
• Client API: searchConversationMessages() (packages/ui/src/api/client-chat.ts:334-337).
• Jump plumbing: ?around= window load (useDataLoaders.ts:451-481, #9955).
• UI: MessageSearchPanel (packages/ui/src/components/chat/message-search/MessageSearchPanel.tsx:33) — mounted only inside ConversationsSidebar:971, which is mounted only in the desktop detached shell (DetachedShellRoot.tsx:157). On web/mobile — the MVP surfaces — search is unreachable (an architecture rule-10 violation: an endpoint without a client trigger on the shipping surface).

Mic + '+' buttons (req 5)

SoftButton (ContinuousChatOverlay.tsx:309-370): borderless 44×44 hit target (WCAG 2.5.5), lucide icons at 26px, hand-drawn glyphs at 30px — the icons were already sized up when chrome was removed; the 26px-vs-30px mix is a minor optical inconsistency (plus glyph renders larger than mic/send). Row inset px-2 py-2 gap-1.5 (:4233).

Pulse matrix (verified):

Surface	While mic hot	Ref
Grabber bar	✅ animate-pulse bg-accent	:509-513, glow at :3703
Collapsed pill	✅ same pulse (glow={listening || responding})	:582-586, :4445
Overlay mic button	❌ static text-accent only	:349-355, :4400-4427
ChatSurface mic (kiosk path)	✅ active && "animate-pulse"	glass-composer.tsx:108
HomePill (foundation path)	✅ animate-pulse bg-warn/70 on listening	HomePill.tsx:66-68

So req 5's pill-pulses-when-collapsed is done; the mic button itself not pulsing on the primary surface is the gap, and it is inconsistent with the sibling path.

Jank sources + perf gating

• The height animation is flexBasis — layout runs every frame by design (the pointer-driven MotionValue avoids re-renders but not reflow; :3985-3995). Rows are React.memo'd with a comparator (chat-message.tsx:379,409); token streaming is context-isolated (useShellController.ts:317-325). No content-visibility/containment anywhere (grep: zero hits) — every open sheet re-lays-out all ≤80 rows per animation frame.
• Hard perf gates exist and run in CI: run-chat-perf-gate.mjs (dropped-frame ratio, p95 frame time, CLS ≤ 0.1 on the REAL overlay) in chat-shell-gestures.yml:160; frame-glitch outlier detection (run-chat-sheet-frame-glitch-e2e.mjs) at :169-175.
• Known red: run-perf-gate-e2e.mjs has a pre-existing CLS failure — the fixture layout-shifts 0.80 during scroll+swipe, reproducing on develop (chat-shell-gestures.yml:18-22) — yet test.yml:333 still runs it as a hard-fail leg. Untriaged.

Test infrastructure (req 6) — strong core, stale edges

Strong: 11 real-browser gesture runners (packages/ui/src/components/shell/__e2e__/), a CHAT_GESTURE_COVERAGE gate that fails CI when a gesture-handler site lacks a matrix row (packages/app/test/chat-gesture-coverage.test.ts, docs in packages/app/docs/CHAT_GESTURE_COVERAGE.md), WebKit engine coverage for the scroll chain, and a canary mode proving the glitch detectors fire.

Stale (post-#13531): chat-clear-swipe.spec.ts:386,424,497 and chat-send-voice-newchat-fuzz.spec.ts:325,382 still drive the removed chat-full-clear button; walkthrough/journey.ts:1108-1121 still expects chat-full-maximize; chat-clear-swipe.spec.ts is still in a CI lane (scenario-pr.yml:341). The unit suite already pins those testids as ABSENT (ContinuousChatOverlay.test.tsx:2308-2309) — the repo is testing both sides of the same contradiction. Minor slop: literal " " filler class strings in cn() calls (:497, :570, HomePill.tsx:54).

Design considerations

• Chat is load-bearing for LifeOps. A parent checking a child's reminder history, or an ADHD user scrolling back to "what did I agree to yesterday", hits the pagination wall immediately. History reachability (scroll-up + search) is MVP-critical, not polish.
• Never fight the scroll. The one thing that most degrades trust in a chat surface is an axis surprise. The fix must be at the scroller (one line + child audit), not per-widget whack-a-mole.
• The gesture system is done — protect it, don't touch it. The detent physics match the brief (lerp on drag, springs on release, half detent, pull-to-top maximize). New work must ride the existing perf gates so nothing slows the drag path.
• Pagination must not regress the drag path. Prepending hundreds of rows into a scroller whose height animates per-frame is the main risk; render windowing and scroll-anchor math must be part of the same change.
• Doctrine: no side panels, no suggestion chips, uniform top bar. Search therefore lives inside the sheet (header affordance at half+), not as a sidebar. Orange accent only; pulse animations use the existing animate-pulse + motion-reduce:animate-none pattern.

Open questions → answers

Q1. Pagination: cursor design and interaction with the 80-row render cap? A: Add ?before=<messageId> to GET messages (pivot's createdAt pushed into the store as end + orderBy desc + limit 100 — identical mechanics to the existing around-window, conversation-routes.ts:1088-1098). Client keeps a hasOlder flag (last page full ⇒ true). Overlay: top sentinel (IntersectionObserver inside #continuous-thread) triggers the fetch; prepend with manual scroll-anchor compensation (record scrollHeight before, restore scrollTop + delta after — overflow-anchor is not implemented in WebKit). The render cap becomes a sliding window over loaded messages (grow by page, bounded ~400) rather than a hard slice(-80); verify with the existing chat-perf-gate thresholds. Rationale: reuses every mechanism already in the file; no virtualizer dependency.

Q2. Where does search live under the no-side-panels doctrine, and what scope? A: A search icon in the sheet header's intentionally-empty left slot (:3933-3935), visible at half+ like the launcher button, opening an in-sheet search row that reuses MessageSearchPanel (it is already presentation-only/injectable). Scope: keep the endpoint's cross-conversation reach; a hit in the active thread jumps via the existing around-window + scroll-into-view; a hit in another conversation switches the active conversation through the existing handleSelectConversation handoff — search becomes the one sanctioned way to reach an older thread now that the switcher is gone. Undecidable-without-owner: whether cross-thread jump violates the one-infinite-thread doctrine. Default: ship active-thread jump first; gate cross-thread jump behind the same PR only if the owner confirms.

Q3. Should the maximize button come back? A: No. #13531's pull-to-maximize is implemented, unit-pinned, and e2e-covered. The remaining work is deleting the stale specs that still look for the button (Q6/issue 5).

Q4. Which states pulse the mic button? A: Pulse exactly when the grabber/ pill glow (listening || responding drives those); for the button itself, pulse on recording || handsFree || transcriptionMode — the same predicate that sets active today (:4420) — so color and motion never disagree. Respect motion-reduce:animate-none (pattern at :512).

Q5. Hide or design horizontal scrolling for wide content? A: Both, split by ownership: the transcript gets overflow-x-hidden (parity with chat-thread-layout.tsx:106); wide content scrolls inside its own row (overflow-x-auto overscroll-x-contain, the MessageContent.tsx:572 pattern) — per repo doctrine "wide content must scroll inside its own container". MessageAttachments.tsx:831 additionally needs overscroll-x-contain.

Q6. Bring ChatSurface/kiosk and detached-window paths to parity? A: Not for MVP. They are desktop/kiosk-only surfaces; the MVP surfaces are web + mobile where ContinuousChatOverlay is the only path. Keep the render-parity contract green; take no feature work there.

Q7. Virtualize long threads? A: No virtualizer for MVP. Measured-first: add per-row content-visibility: auto + contain-intrinsic-size only if the chat-perf-gate numbers regress after pagination lands (issue 6 pairs the CLS triage with this). A virtualizer would fight the mt-auto bottom-anchored layout and the detent height animation for little MVP gain.

Recommendation (minimal-scope MVP plan, ordered)

1. P0 — Lock the transcript to vertical scrolling (1-line scroller fix + child audit + wheel/deltaX regression e2e). Smallest change, biggest feel improvement.
2. P0 — Infinite scroll up with before-cursor pagination (server cursor, client prepend + anchor, sliding render window, perf-gate verification).
3. P1 — Wire message search into the overlay (header affordance reusing MessageSearchPanel + around-window jump; active-thread scope first).
4. P1 — Mic pulse + composer icon optics (pulse on the primary surface, normalize 26/30px optical sizing, keep 44×44 targets; pill pulse already done — add the regression test).
5. P1 — Reconcile stale gesture specs with #13531 (delete/rewrite chat-full-clear/chat-full-maximize consumers; delete the dead verticalScrollPriority option and filler class strings).
6. P2 — Triage the perf-gate CLS 0.80 failure and, only if pagination regresses frame budgets, add row containment.

Everything else in the brief (detents, springs, pull-to-maximize, pill morph, haptics, pill pulse, gesture CI) is already implemented and gated — the job is to not break it, which the existing chat-shell-gestures.yml lane enforces.

Out of scope (explicit non-goals for MVP)

• Feature work on ChatSurface/kiosk/detached-window paths (parity contract only). No consolidation of the three render paths.
• Thread virtualization; framer-motion replacement; transform-based sheet animation rework.
• Cross-conversation UI beyond search-jump (no switcher/swipe resurrection).
• Semantic/vector message search (keyword ILIKE + BM25 is enough).
• New gesture surfaces (e.g. pull-from-transcript detent control — the grabber stays the single drag authority; note: any future scroller-attached pull gesture must use touch events + non-passive preventDefault, since touch-action: pan-y compositor scrolling pointercancels pointer-based gestures).
• Suggestion strips (SHOW_PROMPT_SUGGESTIONS stays off, :223).

Proposed issues

1. [chat-ux] Lock the chat transcript to vertical-only scrolling (P0)
2. [chat-ux] Infinite scroll up: async history pagination in the continuous chat overlay (P0)
3. [chat-ux] Wire message search into the primary chat surface (P1)
4. [chat-ux] Mic button pulses while recording; normalize composer icon optics (P1)
5. [chat-ux] Remove stale gesture specs and dead gesture options left behind by the one-infinite-thread change (P1)
6. [chat-ux] Triage the perf-gate CLS failure and guard long-thread scroll performance (P2)